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ABSTRACT 


The  purpose  of  this  projeet  is  to  analyze  the  Air  Foree’s  usage  rate  of  the  Federal 
Aequisition  Regulation  (FAR)  Subpart  13.5  Test  Program  for  Certain  Commereial  Items. 
FAR  13.500(b)  requires  eontraeting  officers  to  maximize  the  use  of  the  test  program 
when  practicable.  In  addition  to  the  FAR  mandate,  the  Government  Accountability 
Office  (GAO),  found  it  is  in  the  government’s  best  interest  to  use  the  program.  According 
to  a  GAO  report  conducted  in  February  of  2014,  titled  Commercial  Item  Test  Program 
Beneficial,  but  Actions  Needed  to  Mitigate  Potential  Risks,  the  program  improved 
contract  lead-time  and  reduced  required  administration  without  an  increase  in  overall  risk 
to  the  government.  Therefore,  underutilization  of  the  program  will  identify  inefficiencies 
in  the  procurement  process.  This  research  seeks  to  use  the  Federal  Procurement  Data 
System-Next  Generation  (FPDS-NG)  data  to  identify  the  usage  rate  of  the  FAR  Subpart 
13.5  Test  Program  for  Certain  Commercial  Items  in  the  Air  Force  and  provide  potential 
recommendations  to  increase  and  improve  the  test  program’s  use. 
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I.  INTRODUCTION 


A,  BACKGROUND 

On  January  1,  1997,  the  Federal  Aequisition  Reform  Act  (FARA)  temporarily 
authorized  the  application  of  simplified  acquisition  procedures  for  certain  commercial 
items  up  to  $5  million.  Almost  18  years  later,  the  only  substantive  change  to  the  program 
is  an  increased  threshold  of  $6.5  million.  Today,  the  authority  for  the  program  still 
remains  temporary  and  is,  once  again,  due  to  expire  on  January  1,  2015,  unless  Congress 
renews  the  act  or  makes  it  permanent. 

The  importance  of  the  FAR  Subpart  13.5  Test  Program  for  Certain  Commercial 
Items  program  can,  in  part,  be  traced  back  to  a  different  dynamic  for  investments  in 
technology.  In  a  report  titled  Military  R&D  and  Innovation,  David  Mowery  pointed  out 
that  since  the  mid-1980s,  industry  has  outpaced  the  federal  government  in  the  amount  of 
capital  invested  in  research  and  development  (R&D)  projects  (as  cited  in  Gansler,  2014). 
As  a  result,  commercial  technologies  have  become  the  new  standard  for  what  is  state  of 
the  art  and  are  able  to  fulfill  more  government  requirements.  Congressional  legislation 
was  then  passed  to  establish  a  clear  preference  for  commercial  items,  now  codified  in  the 
Federal  Acquisition  Regulations  (FAR,  2014)  under  FAR  Part  12,  Acquisition  of 
Commercial  Items.  The  increasing  relevance  of  commercial  items  in  government 
acquisitions  only  bolsters  the  importance  and  value  of  any  available  procedures,  such  as 
the  test  program,  used  to  procure  those  commercial  goods  and  services. 

Another  dynamic  worth  considering  regarding  the  test  program  is  the  changing 
federal  acquisition  workforce.  As  the  Government  Accountability  Office  (GAO,  2013) 
noted,  “The  acquisition  workforce  plays  a  key  role  in  managing  programs  and  overseeing 
contracts  to  help  agencies  get  what  they  need,  at  the  right  time,  and  at  a  reasonable  price” 
(p.  1);  however,  the  government  is  experiencing  unique  challenges  impacting  the 
retention  of  its  current  employees.  In  2012,  the  Department  of  Defense  (DOD) 
acquisition  workforce  “decreased  for  the  first  time  since  FY  2004.  The  decrease  [was] 
due  to  a  higher  attrition  rate,  including  retirements,  and  ...  fewer  new  hires”  (Federal 
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Acquisition  Institute  [FAI],  2013,  p.  17).  Because  of  increasing  budgetary  pressures 
resulting  from  the  public’s  support  for  austerity  measures  and  an  ever-expanding  number 
of  eligible  retirees,  the  acquisition  workforce  should  be  expected  to  become  smaller  and 
less  experienced.  Appropriately,  these  professionals  will  need  the  right  tools  and 
procedures  to  enhance  their  productivity.  With  such  concerns  in  mind,  FAR  Subpart  13.5 
was  established  to  increase  efficiencies  and  alleviate  administrative  burdens  for 
contracting  organizations  through  expanding  the  usefulness  and  applicability  of 
simplified  acquisition  procedures. 

B,  RESEARCH  OBJECTIVES 

The  main  objective  of  this  research  paper  is  to  examine  the  Air  Force’s  usage  of 
the  FAR  Subpart  13.5  Test  Program  for  Certain  Commercial  Items,  specifically  from 
February  1,  2013  through  May  31,  2014,  to  determine  whether  contracting  offices  are 
employing  “the  simplified  procedures  authorized  by  the  test  to  the  maximum  extent 
practicable”  (FAR  13.500[b]).  Furthermore,  this  research  attempts  to  identify  and 
investigate  various  metrics  in  an  effort  to  analyze  the  relative  effectiveness  of  the  test 
program  procedures.  Finally,  this  report  makes  recommendations  on  how  to  increase  the 
usage  and  effectiveness  of  the  test  program.  More  specifically,  this  report  answers  the 
following  questions: 

1 .  What  was  the  Air  Force’s  usage  rate  of  FAR  13.5  Test  Program  for 
Certain  Commercial  Items  procedures  from  February  1,  2013,  through 
May  31,2014? 

2.  What  factors  may  be  limiting  the  use  of  FAR  13.5  Test  Program  for 
Certain  Commercial  Items  procedures? 

C.  ORGANIZATION 

Chapter  I  provides  a  brief  discussion  on  the  background  of  the  FAR  13.5  Test 
Program  for  Certain  Commercial  Items,  lists  the  research  objectives,  and  explains  the 
benefits  of  this  report. 

Chapter  II  provides  a  literature  review  of  the  pertinent  acquisition  regulations  and 
research  conducted  to  date.  The  first  section  provides  a  history  of  FAR  Subpart  13.5 
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through  an  examination  of  acquisition  reform  initiatives,  Federal  Acquisition 
Streamlining  Act  of  1994  (FAS A)  and  the  Federal  Acquisition  Reform  Act  of  1996 
(FARA)  as  well  as  the  FAR  Subpart  13.5  policies  and  procedures.  The  second  section  of 
this  chapter  discusses  the  importance  and  benefits  of  commercial  item  and  simplified 
acquisition  procedures  and  examines  previous  studies  on  FAR  13.5  usage  rate.  The  last 
section  provides  a  chronological  review  of  the  development  of  the  Federal  Procurement 
Data  System-Next  Generation  (FPDS-NG)  and  the  reports  regarding  its  accuracy. 

Chapter  III  explains  the  methodology,  scope,  and  limitations  in  a  step-by-step 
manner  that  is  designed  to  allow  replication  and  improvements  using  additional  data  sets. 

Chapter  IV  presents  the  FPDS-NG  data  and  provides  analysis  of  the  FAR  13.5 
test  program  usage  rate,  dollar  value  distribution,  competition  levels,  number  of  offerors, 
and  possible  factors  contributing  to  use  or  non-use  of  the  FAR  13.5  test  program. 

Chapter  V  summarizes  the  report’s  findings  by  answering  the  research  questions 
and  provides  recommendations  for  improving  the  usage  rate  of  the  test  program.  It  also 
provides  recommendations  for  future  research  studies. 

D,  BENEFITS  OF  THE  REPORT 

This  report  benefits  federal  agencies,  contracting  offices,  contractors,  small 
business.  Congress,  and  ultimately  U.S.  taxpayers.  This  report  explains  the  benefits  of 
reduced  administrative  burden  and  cost  savings  associated  with  using  the  commercial 
item  test  program.  The  reduced  workload  directly  benefits  contracting  offices  and 
contractors,  allowing  them  to  reduce  manpower  and  focus  efforts  on  more  valuable 
functions.  This  manpower  and  workload  reduction  translates  into  cost  savings  for 
companies  and  federal  agencies,  which  can  be  passed  on  to  the  U.S.  taxpayer.  An 
accurate  usage  rate  provides  Congress  with  a  sound  data  point  necessary  to  make  the 
FAR  13.5  test  program  permanent.  They  can  save  time  and  effort  associated  with  the 
regularly  renewal  of  the  program  and  provide  a  stable  tool  for  the  contracting  offices  to 
use.  Finally,  additional  acquisitions  using  simplified  acquisition  procedures  opens  up 
more  opportunities  for  small  businesses  that  are  not  capable  of  or  willing  to  be  involved 
in  a  large  and  complicated  source  selection  process. 
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E,  CHAPTER  SUMMARY 

This  chapter  discussed  the  background  of  the  FAR  Subpart  13.5  Test  Program  for 
Certain  Commereial  Items  by  explaining  the  aequisition  reform  legislation  leading  to  the 
program.  It  also  deseribed  the  researeh  objeetives  by  presenting  two  usage  rate  questions. 
The  organization  of  the  paper  was  then  presented  by  explaining  each  of  the  contents  of 
eaeh  chapter.  Lastly,  the  benefits  of  this  paper  to  federal  ageneies,  eontraeting  offiees, 
contraetors,  small  business,  Congress,  and  taxpayers  were  provided. 
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II.  LITERATURE  REVIEW 


A,  CHAPTER  INTRODUCTION 

The  purpose  of  this  ehapter  is  to  provide  a  review  of  the  literature  directly  related 
to  the  FAR  Subpart  13.5  Test  Program  for  Certain  Commercial  Items  and  the  accuracy  of 
FPDS-NG.  A  review  of  the  program’s  history  will  discuss  acquisition  reform  legislation 
and  the  resulting  FAR  13.5  policies  and  procedures.  This  discussion  will  be  followed  by 
an  explanation  of  four  conceptual  program  benefits  and  a  review  of  previous  studies 
aimed  at  determining  benefits.  Next,  a  chronological  review  of  usage  rate  studies  since 
2001  will  be  reviewed  to  examine  various  methodologies  and  resulting  usage  rates. 
Lastly,  the  history  of  events  leading  to  the  current  state  of  FPDS-NG  will  be  provided  as 
well  as  past  studies  to  discuss  FPDS’  accuracy  and  reliability  improvements. 

B,  HISTORY  OF  FAR  SUBPART  13.5,  THE  TEST  PROGRAM  FOR 
CERTAIN  COMMERCIAL  ITEMS 

1.  Acquisition  Reform:  The  Federal  Acquisition  Streamlining  Act  of 
1994  and  Federal  Acquisition  Reform  Act  of  1996 

In  response  to  years  of  unsuccessful  attempts  to  simplify  the  acquisition  process 
and  reduce  the  excessive  use  of  unique  government  specifications,  the  National  Defense 
Authorization  Act  (NDAA)  for  Fiscal  Year  (FY)  1991  directed  the  Department  of 
Defense  (DOD)  to  establish  what  would  become  known  as  the  Section  800  Panel  (GAO, 
1993).  This  panel  was  tasked  to 

(1)  review  the  acquisition  laws  applicable  to  DOD  with  a  view  toward 
streamlining  the  defense  acquisition  process;  (2)  make  any 
recommendations  for  the  repeal  or  amendment  of  such  laws  as  the  Panel 
considered  necessary;  and  (3)  prepare  a  proposed  code  of  relevant 
acquisition  laws.  (GAO,  1993,  p.  3) 

Over  the  span  of  two  years,  the  Section  800  Panel  reviewed  over  600  laws, 
ultimately  producing  a  1,800-page  report  containing  more  than  300  recommendations 
focused  on  “formulating  changes  to  streamline  and  simplify  the  defense  acquisition 
process  and  to  improve  the  DOD’s  capability  to  purchase  commercial  items  and 
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technologies”  (GAO,  1993,  p.  3).  Many  of  those  recommendations  served  as  the  impetus 
for  legislative  reforms  in  the  years  to  follow. 

The  Federal  Acquisition  Streamlining  Act  of  1994  (FAS A)  and  the  Federal 
Acquisition  Reform  Act  of  1996  (FARA)  were  two  pieces  of  legislation  that  leveraged 
the  recommendations  of  the  Section  800  Panel  to  establish  a  foundation  for  increasing  the 
acquisition  of  commercial  items  to  meet  the  federal  government’s  needs.  Signed  into  law 
a  decade  after  the  passage  of  the  Competition  in  Contracting  Act  (CICA),  which  reshaped 
the  competitive  landscape  of  the  acquisition  community,  the  FASA’s  and  FARA’s 
reforms  centered  around  reducing  requirements  and  procedures  within  the  acquisition 
process  that  were  considered  too  costly,  complex,  and  burdensome. 

The  comprehensiveness  of  the  FASA  is  best  demonstrated  by  its  sheer  magnitude 
and  breadth.  The  legislation  totaled  167  pages  and  directed  changes  spanning  across 
almost  every  conceivable  acquisition  area,  specifically  Contract  Formation  (Title  1), 
Contract  Administration  (Title  11),  Service  Specific  and  Major  Systems  Statutes  (Title 
111),  the  Simplified  Acquisition  Threshold  (SAT;  Title  IV),  Acquisition  Management 
(Title  V),  Other  Procurement-Related  Matters  (Title  VI),  Small  Business  and 
Socioeconomic  Laws  (Title  Vll),  Commercial  Items  (Title  Vlll),  and  even  establishing  a 
Federal  Acquisition  Computer  Network  (Title  IX;  FASA,  1994).  In  particular,  the 
requirements  of  Title  IV,  Simplified  Acquisition  Threshold,  and  Title  VIII,  Commercial 
Items,  provided  significant  reductions  to  the  complexity  of  the  federal  acquisition  system 
while  also  increasing  its  overall  responsiveness. 

The  FASA’s  Title  IV  reforms  to  the  SAT  streamlined  government  acquisitions 
significantly.  Although  there  were  a  number  of  legislative  requirements  under  Title  IV, 
the  most  notable  change  was  the  large  increase  to  the  SAT,  which  raised  the  ceiling  from 
$25,000  to  $100,000.  In  1993,  the  DOD  reported  that  “the  lead  time  for  awards  less  than 
$25,000  [was]  26  days  [while]  the  lead  time  for  awards  above  $25,000  [was]  90  days  for 
simple  sealed  competitive  bids  and  210  days  for  negotiated  competitive  bids”  (DOD, 
1993,  p.  4).  Furthermore,  the  report  noted  that  a  $100,000  threshold  would  allow  the 
DOD  to  accomplish  an  estimated  99%  of  all  contract  actions  using  simplified  acquisition 

procedures  (DOD,  1993).  Naturally,  through  dramatically  expanding  the  number  of 
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acquisitions  under  the  SAT,  the  FASA’s  reforms  would  drastieally  improve  the  speed  and 
effieieney  of  the  government’s  aequisition  proeess. 

Regarding  inereasing  the  aequisition  of  eommereial  items  under  Title  VIII,  the 
FASA  ehanged  the  manner  in  whieh  the  government  eondueted  routine  business  through 
numerous  measures  that  aligned  federal  proeurements  with  industry  praetiees.  First,  the 
FASA  established  eomprehensive  definitions  for  the  terms  eommereial  item  and 
nondevelopmental  item  to  govern  the  applieability  of  the  new  statutes.  Then,  the  FASA 
sought  to  reduee  the  eost  and  eomplexity  for  eompanies  doing  business  with  the 
government  by  limiting  the  elauses  applieable  to  any  eontraets  for  eommereial  items  and 
by  exempting  sueh  aequisitions  from  other  miseellaneous  statutory  requirements.  Most 
importantly,  however,  the  FASA  established  the  “Preferenee  for  Aequisition  of 
Commereial  Items”  under  Seetion  8104,  whieh  required  the  government  to  proaetively 
souree  eommereial  items  by  mandating,  “to  the  maximum  extent  praetieable, 
[requirements  be  defined]  so  that  eommereial  items,  or  to  the  extent  that  eommereial 
items  suitable  to  meet  the  ageney’s  needs  are  not  available,  nondevelopmental  items 
other  than  eommereial  items,  may  be  proeured  to  fulfill  sueh  requirements”  (FASA, 
1994,  §  2377(a))  and  that  proeurement  offieials  “aequire  eommereial  items  or 
nondevelopmental  items  other  than  eommereial  items  to  meet  the  needs  of  the  ageney” 
(FASA,  1994,  §  2377(b)).  By  altering  the  way  ageneies  define  requirements,  requiring 
market  researeh,  and  mandating  proeurement  offieials  seek  eommereial  goods  and 
serviee  solutions  first,  the  FASA  moved  federal  aequisitions  mueh  eloser  to  a  market- 
based  system. 

Two  years  later,  the  FARA  was  enaeted  into  law  as  part  of  the  NDAA  for  FY 
1996  under  Division  D  to  serve  as  a  logieal  eontinuation  of  the  FASA’s  reforms,  foeusing 
partieular  attention  on  eompetition  requirements  and  eommereial  item  aequisitions 
(FARA,  1996).  While  the  FARA  eontained  a  number  of  notable  reforms  to  the  federal 
aequisition  system,  one  partieular  reform  sought  to  expand  on  the  sueeess  of  both  the 
inereased  SAT  and  the  new  preferenees  for  eommereial  items.  Seetion  4202,  titled 
Applieation  of  Simplified  Proeedures  to  Certain  Commereial  Items,  amended  Seetion 
2304(g)  of  Title  10,  United  States  Code,  to  establish  the  authority  for  the  Test  Program 
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for  Certain  Commercial  Items.  At  the  time,  this  program  temporarily  authorized  the  use 
of  the  highly  successful  Simplified  Acquisition  Procedures  on  requirements  for 
commercial  supplies  and  services  up  to  $5  million.  The  law  was  incorporated  into  the 
FAR  under  Subpart  13.5  and  titled  Test  Program  for  Certain  Commercial  Items. 
Established  as  a  temporary  program,  the  test  program  has  generally  been  renewed  every 
two  years  with  some  lapses  in  coverage.  Most  recently,  in  2012,  Congress  allowed  the 
test  program  to  expire  for  a  brief  period  before  extending  it  again  a  year  later  until 
January  1,  2015,  as  part  of  the  NDAA  for  FY  2013  (Defense  Procurement  and 
Acquisition  Policy  [DPAP],  2012). 

2,  FAR  Subpart  13,5  Policies  and  Procedures 

When  active,  the  Test  Program  for  Certain  Commercial  Items  under  FAR  Subpart 
13.5,  permits  the 

use  of  simplified  acquisition  procedures  for  the  acquisition  of  supplies  and 
services  in  amounts  greater  than  the  simplified  acquisition  threshold  but 
not  exceeding  $6.5  million  ...  including  options,  if  the  contracting  officer 
reasonably  expects,  based  on  the  nature  of  the  supplies  or  services  sought, 
and  on  market  research,  that  offers  will  include  only  commercial  items. 

(FAR  13.500(a)) 

In  order  to  fully  comprehend  the  potential  impacts  of  the  test  program,  this  paper 
reviews  what  Simplified  Acquisition  Procedures  are  and  how  they  differ  from  other 
acquisition  methods. 

FAR  Part  13  prescribes  the  policies  and  guidelines  governing  the  use  of 
Simplified  Acquisition  Procedures.  More  specifically,  FAR  Subpart  13.1,  Procedures, 
and  FAR  Subpart  13.3,  Simplified  Acquisition  Methods,  contain  a  number  of  unique 
methods  for  contracting  actions  utilizing  Simplified  Acquisition  Procedures.  Those 
methods  include  the  ability  to 

•  solicit  quotes  as  well  as  offers  (FAR  13.106), 

•  use  standing  price  quotations  (FAR  13.103), 

•  establish  and  order  from  blanket  purchase  agreements  (FAR  13.303), 
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•  “use  any  appropriate  eombination  of  the  proeedures  in  Parts  13,  14,  15,  35, 
or  36”  (FAR  13.003(g)),  and 

•  “use  innovative  approaches,  to  the  maximum  extent  practicable,  in 
awarding  contracts  using  simplified  acquisition  procedures”  (e.g.,  reverse 
auction  procedures;  FAR  13.003(h)(4)). 

Regarding  the  contract  file  documentation  for  acquisitions  conducted  under  the 
test  program,  the  FAR  states, 

the  contract  file  must  include  (1)  a  brief  written  description  of  the 
procedures  used  in  awarding  the  contract,  including  the  fact  that  the  test 
procedures  in  FAR  Subpart  13.5  were  used;  (2)  the  number  of  offers 
received;  (3)  an  explanation,  tailored  to  the  size  and  complexity  of  the 
acquisition,  of  the  basis  for  the  contract  award  decision;  and  (4)  any 
justification  approved  under  paragraph  (a)  of  this  section.  (FAR  13.501(b), 

2014) 

Finally,  the  FAR  requires  that  “for  the  period  of  this  test,  contracting  activities 
must  employ  the  simplified  procedures  authorized  by  the  test  to  the  maximum  extent 
practicable”  (FAR  13.500(b),  2014).  The  use  of  the  term  “maximum  extent  practicable” 
versus  the  clear  mandate  that  would  have  been  imposed  by  using  the  term  “shall” 
indicates  that  contracting  activities  have  a  degree  of  flexibility  when  it  comes  to  the  use 
of  the  test  program.  Some  circumstances  that  might  reasonably  justify  foregoing  the  use 
of  the  Simplified  Acquisition  Procedures  authorized  by  the  program  can  be  found  under 
FAR  13.003(a),  which  include 

•  sources  of  supply  under  FAR  Part  8, 

•  existing  indefinite  delivery/indefinite  quantity  contracts,  or 

•  other  established  contracts. 

Such  exceptions  to  the  test  program  make  sense  because  they  fall  into  either  the 
category  of  mandatory  use  (e.g..  Federal  Prison  Industries  [FPl]  under  FAR  Subpart  8.6, 
Nonprofit  Agencies  Employing  People  Who  Are  Blind  or  Severely  Disabled  under  FAR 
Subpart  8.7,  Requirements  Contracts  under  FAR  16.503)  or  a  method  that  already 
provides  a  comparatively  streamlined  acquisition  process  (e.g..  Federal  Supply  Schedules 
under  FAR  Subpart  8.4  and  Indefinite-Delivery  Contracts  under  FAR  Subpart  16.5); 
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however,  aequisitions  that  seemingly  forego  the  use  of  the  Test  Program  for  Certain 
Commereial  Items  without  fulfilling  the  aforementioned  criteria  likely  subject  themselves 
needlessly  to  a  more  risky,  complex  process  with  larger  procedural  limitations. 


C.  IMPORTANCE/BENEFITS  OF  FAR  SUBPART  13.5 

The  potential  advantages  of  FAR  Subpart  13.5  can  be  traced  back  to  the  general 
scope  of  FAR  Part  13,  which  states. 

The  purpose  of  this  part  is  to  prescribe  simplified  acquisition  procedures 
in  order  to  (a)  Reduce  administrative  costs;  (b)  Improve  opportunities  for 
small,  small  disadvantaged,  women-owned,  HUBZone,  and  service- 
disabled  veteran-owned  small  business  concerns  to  obtain  a  fair  proportion 
of  Government  contracts;  (c)  Promote  efficiency  and  economy  in 
contracting;  and  (d)  Avoid  unnecessary  burdens  for  agencies  and 
contractors.  (FAR  13.002,  2014) 

It  stands  to  reason  then  that  the  objective  for  the  Test  Program  for  Certain 
Commercial  Items  is  to  accomplish  those  same  objectives,  but  for  commercial  item 
acquisitions  in  excess  of  $150,000  up  to  $6.5  million.  Unfortunately,  as  the  GAO  (2014) 
found,  “data  are  not  specifically  collected  to  assess  test  program  benefits  on  the  basis  of 
such  metrics  as  contracting  lead  time”  (p.  14).  Therefore,  in  order  to  assess  the 
importance  of  the  test  program,  this  paper  discusses  the  benefits  that  FAR  Subpart  13.5 
provides  from  a  conceptual  standpoint  and  then  reviews  any  findings  from  previous 
studies. 


1.  Conceptual 

If  utilized  correctly,  the  test  program  should  theoretically  allow  agencies  to 
achieve  the  four  previously  identified  benefits  of  simplified  acquisition  procedures 
(reduced  administrative  costs,  improved  small  business  opportunities,  greater 
efficiencies,  and  reduced  burdens),  but  on  a  broader  scale.  This  section  reviews  each  of 
those  objective  areas  and  how  the  procedures  authorized  by  FAR  Subpart  13.5  can 
uniquely  contribute  to  each  one. 
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a.  Benefit  #1-Reduced  Administrative  Costs 

First,  the  test  program  reduces  administrative  costs  by  minimizing  documentation 
and  prioritizing  the  efficiency  of  the  acquisition  process.  In  regard  to  reducing  costs  by 
limiting  contract  file  documentation,  FAR  13.106-3(b)  instructs  acquisition  officials  to 
“[k]eep  documentation  to  a  minimum.  Purchasing  offices  shall  retain  data  supporting 
purchases  (paper  or  electronic)  to  the  minimum  extent  and  duration  necessary  for 
management  review  purposes  (see  subpart  4.8).”  As  previously  mentioned,  the  specific 
documentation  requirements  for  the  test  program  are  merely  limited  to  four  components 
consisting  of 

a  brief  description  of  the  procedures  used  in  awarding  the  contract,  ...  the 
number  of  offers  received,  ...  an  explanation,  tailored  to  the  size  and 
complexity  of  the  acquisition,  of  the  basis  for  the  contract  award  decision; 
and  [if  necessary]  ...  any  justification  [for  sole  source  or  brand  name 
acquisitions].  (FAR  13.501,  2014) 

Another  way  the  test  program  minimizes  administrative  costs  is  through 
encouraging  efficiency  throughout  the  acquisition  process.  During  the  solicitation  phase, 
simplified  acquisition  procedures  instruct  agencies  to  “respond  to  inquiries  received 
through  any  medium  (including  electronic  commerce)  if  doing  so  would  not  interfere 
with  the  efficient  conduct  of  the  acquisition”  (FAR  13.106-l(f),  2014).  Simplified 
acquisition  procedures  provide  further  efficiencies  during  the  evaluation  phase  by 
requiring  contracting  officers  to  “ensure  that  quotations  or  offers  can  be  evaluated  in  an 
efficient  and  minimally  burdensome  fashion”  (FAR  13.106-2(b)(3),  2014).  By 
encouraging  (even  requiring)  efficiency  when  utilizing  simplified  acquisition  procedures 
under  the  test  program,  contracting  officers  have  a  unique  basis  for  supporting  the 
reasonableness  of  decisions  to  eliminate  any  inefficiencies  in  both  phases  of  the 
acquisition  process,  thereby  reducing  administrative  costs. 

b.  Benefit  #2-Improved  Small  Business  Opportunities 

Unlike  acquisitions  under  the  SAT,  acquisitions  conducted  under  the  authority  of 
the  Test  Program  are  not  reserved  exclusively  for  small  business  concerns  (FAR 
13.003(b)(1)).  Still,  the  use  of  simplified  acquisition  procedures  seeks  to  increase  the 
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opportunities  for  small  businesses  to  partieipate  in  government  eontraets  through 
redueing  the  eomplexity  of  the  eontracting  proeess.  Solicitations  developed  in  accordance 
with  FAR  Part  13  procedures  are  generally  shorter,  simpler,  and  more  understandable 
than  FAR  Part  14  or  15  solicitations.  Such  differences  ultimately  make  government 
contracting  opportunities  more  accessible,  particularly  for  small  businesses  that  may  lack 
the  staff  and/or  experience  necessary  to  navigate  through  a  more  complex, 
comprehensive  process.  Furthermore,  utilizing  the  test  program  allows  any  interested 
commercial  small  businesses  to  respond  to  solicitations,  whereas  Indefinite  Delivery — 
Indefinite  Quantity  (IDIQ)  contracts  or  General  Service  Administration  (GSA)  schedules 
limit  opportunities  only  to  those  companies  who  have  a  previously  established  contract 
with  the  respective  organization.  Although  such  limitations  are  fair  and  in  accordance 
with  CICA,  they  still  limit  the  potential  pool  of  suppliers  when  compared  to  the  complete 
openness  of  a  combined  synopsis-solicitation  under  the  Test  Program  for  Certain 
Commercial  Items. 

c.  Benefit  #3-Greater  Efficiencies 

In  terms  of  promoting  efficiency  and  economy  in  contracting,  the  test  program 
accomplishes  these  goals  in  two  main  ways.  First,  the  evaluation  procedures  prescribed 
by  FAR  Part  13  allow  contracting  officers  the  ability  to  significantly  streamline  a  process 
that  would  otherwise  be  exhaustive  and  time  consuming  for  acquisitions  in  excess  of  the 
SAT.  Specifically,  FAR  13.106-2(b)(4)  states. 

For  acquisitions  conducted  using  a  method  that  permits  electronic 
response  to  the  solicitation,  the  contracting  officer  may — (i)  After 
preliminary  consideration  of  all  quotations  or  offers,  identify  from  all 
quotations  or  offers  received  one  that  is  suitable  to  the  user,  such  as  the 
lowest  priced  brand  name  product,  and  quickly  screen  all  lower  priced 
quotations  or  offers  based  on  readily  discemable  value  indicators,  such  as 
past  performance,  warranty  conditions,  and  maintenance  availability;  or 
(ii)  Where  an  evaluation  is  based  only  on  price  and  past  performance, 
make  an  award  based  on  whether  the  lowest  priced  of  the  quotations  or 
offers  having  the  highest  past  performance  rating  possible  represents  the 
best  value  when  compared  to  any  lower  priced  quotation  or  offer. 
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Comparing  these  proeesses  to  the  eomprehensive  and  restrictive  procedures  for 
opening  and  evaluating  bids  under  FAR  Subpart  14.4  or  conducting  source  selections 
under  FAR  Subpart  15.3  easily  illustrates  the  efficiencies  afforded  by  the  test  program. 
For  example,  when  utilizing  the  test  program,  contracting  officers  “are  not  required  to 
establish  a  formal  evaluation  plan  or  competitive  range,  conduct  discussions  with 
vendors,  or  score  quotations  from  offerors”  (GAO,  2001,  p.  3),  all  of  which  are  more 
labor  intensive  than  the  procedures  authorized  by  FAR  Part  13. 

A  second  area  where  FAR  Subpart  13.5  promotes  economies  throughout  the 
procurement  process  is  the  ability  to  use  innovative  approaches  when  conducting 
acquisitions  under  the  authority  of  the  test  program.  More  specifically, 

FAR  13.003(g)  gives  wide  latitude  in  the  procedures  that  can  be  used  for 
simplified  acquisitions  by  allowing  “any  appropriate  combination”  of  the 
procedures  in  other  parts  of  the  FAR  and  encouraging  the  use  of 
“innovative  approaches,  to  the  maximum  extent  practicable.”  This  had 
been  held  to  permit  the  use  of  reverse  auction  procedures.  (Cibinic,  Nash, 
&Yukins,  2011,  p.  1040) 

As  stated  within  the  guiding  principles  for  the  Federal  Acquisition  System,  as 
long  as  members  of  the  acquisition  team  exercise 

sound  business  judgment  ...  if  a  specific  strategy,  practice,  policy  or 
procedure  is  in  the  best  interests  of  the  Government  and  is  not  addressed 
in  the  FAR,  nor  prohibited  by  law  (statute  or  case  law).  Executive  order  or 
other  regulation... the  strategy,  practice,  policy  or  procedure  is  a 
permissible  exercise  of  authority.  (FAR  1.102(d),  2014) 

Because  simplified  acquisition  procedures  naturally  have  fewer  restrictions,  they 
uniquely  allow  acquisition  officials  greater  latitude  to  conduct  acquisitions  in  a  manner 
that  is  most  economical  for  the  government. 

d.  Benefit  #4-Reduced  Burdens 

Finally,  the  test  program  avoids  unnecessary  burdens  for  agencies  and  contractors 
alike  through  the  ability  to  issue  requests  for  quotes  (RFQs)  instead  of  invitations  for  bids 
(IFBs)  or  requests  for  proposals  (RFPs),  which  require  interested  parties  to  submit  a 
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binding  offer  in  the  fonn  of  a  bid  or  proposal,  respectively.  The  RFQ  process 
differentiates  itself  from  IFBs  and  RPPs  because 

a  quotation  is  not  an  offer  and,  consequently,  cannot  be  accepted  by  the 
Government  to  fonn  a  binding  contract  ...  [instead],  the  order  is  an  offer 
by  the  Government  to  the  supplier  . . .  [and]  a  contract  is  established  when 
the  supplier  accepts  the  offer.  (FAR  13.004) 

Thus,  RFQs  have  the  imique  benefit  of  providing  “a  substantially  lower  risk  for 
the  supplier  than  that  which  would  be  incmTed  if  the  supplier  were  making  an  offer” 
(Cibuiic  et  al.,  2011,  p.  1036).  Similarly,  “an  RFQ  also  gives  the  agency  more  flexibility 
in  conducting  a  prociuement”  (Cibinic  et  al.,  2011,  p.  1036)  because  the  government  carr, 
for  example,  irse  standing  price  qirotations  as  the  basis  for  a  contract,  which  fiuther 
reditces  the  brrrden  on  the  goverirment.  FruHrernrore,  the  government  is  rrot  boimd  by 
RFQs  to  either  award  a  pruchase  order  or  cancel  the  solicitation,  wliich  fiuther  reduces 
biudeus  on  goveriuuent  agencies  (FAR  15.402(e),  2014).  Still,  if  the  sitiration  warrants 
the  irse  of  an  RFP  or  arr  IFB  to  bind  a  contractor  throirglr  the  sirbmission  of  an  offer,  the 
test  program  allows  the  government  to  irse  those  approaches  as  well.  Table  1  summarizes 
the  benefits  previoirsly  addressed  in  this  section. 


Table  1.  Simmrary  of  FAR  Sirbpart  13.5  Benefits 


1.  Redirced  Administrative  Costs 

Limiting  contract  file  docirmerrtation 

More  efficierrt  acquisitiorr  process 

2.  Improved  Small  Birsmess  Opportrmities 

Shorter,  simpler  solicitations  imder  SAP  are  likely  more  easily  imderstood 

Allows  any  eligible,  interested  company  to  respond  vs.  IDVs,  which  are  exchrsive 

3.  Greater  Efficiencies 

Permits  a  streamlmed  evahration  process 

Allows  the  irse  of  irmovative  approaches,  such  as  reverse  auctions 

4.  Reduced  Biudeus 

Allows  use  of  RFQs,  which  are  generally  less  biudensome 

Still  permits  the  use  of  RFPs  and  IFBs,  if  wananted 

14 


Ultimately,  the  various  policies  and  procedures  governing  the  Test  Program  for 
Certain  Commercial  Items  allows  acquisition  officials  unique  flexibilities  to  be  more 
efficient  and  open  with  procurements.  To  what  extent,  specifically,  is  a  question  that 
previous  studies  have  attempted  to  answer. 

2.  Previous  Studies 

Researchers  have  attempted  to  quantify  the  benefits  of  the  commercial  item  test 
program  since  at  least  1999,  when  “the  Office  of  Federal  Procurement  Policy  (OFPP) 
surveyed  procurement  executives  in  federal  agencies  to  obtain  their  opinions  on  the 
benefits”  (GAO,  2001,  p.  4).  Although  early  GAO  reports  struggled  to  find  empirical  data 
to  determine  the  benefits  of  the  program,  recent  studies  have  shown  the  program’s  worth. 
The  following  is  a  summary  of  the  studies  that  have  been  accomplished  to  date. 

One  of  the  earliest  measures  of  the  commercial  test  program’s  benefits  came  in 
the  form  of  a  survey  conducted  in  1999  by  the  OFPP  (GAO,  2001).  The  OFPP  polled 
government  procurement  executives  and  received  responses  that  claimed  several  believed 
benefits.  The  benefits  included  “a  positive  impact  on  (1)  time  required  to  award  a 
contract,  (2)  administrative  costs,  (3)  prices,  (4)  small  business  participation,  and  (5) 
delivery  of  products  and  services”  (GAO,  2001,  p.  4).  Although  this  initial  report  on  the 
program  was  positive,  the  GAO  (2001)  quickly  pointed  out  some  key  weaknesses  about 
the  survey.  Some  weaknesses  included  that 

the  survey  did  not  measure  the  extent  to  which  (1)  time  required  to  award 
contracts  was  reduced,  (2)  administrative  costs  were  reduced,  (3)  prices 
reflected  the  best  value,  (4)  small  business  participation  was  promoted,  or 
(5)  delivery  of  products  and  services  was  improved.  (GAO,  2001,  p.  4) 

The  GAO’s  ultimate  conclusion  in  the  2001  report  was  that  despite  the  federal 
procurement  executives’  desire  to  make  the  program  permanent,  benefits  of  the  program 
have  not  yet  been  clearly  demonstrated.  The  GAO  recommended  that  Congress  require 
that  the  OFPP  construct  a  new  methodology  of  testing  the  program’s  benefits.  Congress 
responded  by  mandating  that  the  GAO  determine  the  extent  to  which  the  DOD  was  using 
the  program  and  any  resulting  benefits  (GAO,  2003).  The  GAO  issued  a  report  in  2003 
that  stated  although  interviews  with  procurement  officials  showed  favorable  program 

15 


results,  the  GAO  was  unable  to  aeeomplish  either  mandated  task  due  to  a  laek  of  reliable 
data  (GAO,  2003a).  The  GAO  eritieized  the  reliability  of  FPDS  data  and  pointed  out 
federal  ageneies’  laek  of  progress  in  gathering  useable  data  on  the  program. 

In  2006,  a  signifioant  researeh  study  was  eondueted  on  the  Navy  Fleet  and 
Industrial  Supply  Center’s  (FISC)  use  of  the  test  program  (Johnson,  Simonson,  & 
Ziegler,  2006).  The  results  quantified  benefits  using  transaetion  toueh  time  and  ultimately 
determined  a  90%  reduetion  in  time  and  eost  when  using  the  program  (Johnson  et  ah, 
2006).  The  study  found  “an  average  eost  reduetion  of  over  $9,500  per  transaction” 
(Johnson  et  ah,  2006,  p.  21)  when  using  FAR  13.5  instead  of  traditional  protocol.  The 
study  was  one  of  the  first  to  actually  quantify  the  benefits  of  the  test  program. 

In  2008,  the  DOD  provided  a  report  to  Congress  outlining  several  benefits  of  the 
test  program.  These  benefits  included  “increased  competition  and  small  business 
participation,  decreased  acquisition  lead  time,  enhanced  workforce  morale,  and  decreased 
costs  all  while  increasing  the  war  fighter’s  capabilities”  (DOD,  2008,  p.  7).  The  DOD’s 
(2008)  report  went  to  on  to  express  the  department’s  desire  to  make  the  program 
permanent,  stating,  “Incremental  extension  of  the  Test  Program  is  a  barrier  to  increasing 
the  use  of  commercial  items”  (p.  7).  After  the  test  program’s  brief  expiration  in  2012,  the 
DOD  (2012)  provided  another  report  that  stated,  “Expiration  of  the  FAR  13.5  Test 
Program  for  Certain  Commercial  Items  resulted  in  an  additional  barrier  to  competition  for 
actions  greater  than  the  simplified  acquisition  threshold”  (p.  13). 

In  2014,  the  GAO  developed  a  methodology  to  assess  the  test  program  and 
formally  acknowledged  its  benefits  in  a  report  to  Congress.  The  GAO’s  (2014) 
performance  audit  “selected  a  nonrandom,  nongeneralizable  sample”  (p.  2)  from  the 
DOD,  DOI,  and  DHS,  and  determined  that  the  test  program  “reduced  contracting  lead 
time  and  administrative  burdens  and  generally  did  not  incur  additional  risks  above  those 
on  other  federal  acquisition  efforts  for  those  contracts”  (GAO,  2014,  para.  2).  This  report 
was  significant  because  it  represented  the  first  time  the  GAO  offered  a  quantitative 
response  to  Congress’  mandate  regarding  the  test  program’s  benefits. 
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As  discussed  in  previous  paragraphs,  past  GAO  reports  regularly  stated  that 
reliable  data  was  an  issue  in  determining  the  program’s  benefits.  As  explained  in  Seetion 

C,  the  same  ehallenge  existed  in  the  mandate  to  determine  an  aceurate  usage  rate. 

D,  PREVIOUS  USAGE  RATE  STUDIES 

As  early  as  2001,  researchers,  such  as  the  GAO,  have  attempted  to  quantify  the 
use  of  the  FAR  Subpart  13.5,  Test  Program  for  Certain  Commereial  Items,  and  measure 
the  program’s  use.  Studies  on  several  agencies  have  produeed  varied  results.  This 
variation  is  largely  due  to  eaeh  study  using  its  own  methodology  and  data  set.  The 
following  seetion  provides  a  ehronologieal  literature  review  of  the  studies’  methodologies 
and  results.  The  ehronologieal  order  traees  the  ehange  in  researeh  design  and  usage  rates 
throughout  the  last  15  years. 

The  NDAA  for  FY  2000  first  mandated  that  the  GAO  evaluate  the  FAR  Subpart 
13.5,  Test  Program  for  Certain  Commereial  Items  (GAO,  2001).  The  GAO  approaehed 
the  mandate  by  reviewing  12  total  eontraets  that  used  the  test  program  at  the  Air  Foree’s 
Air  Armament  Command  at  Eglin  Air  Foree  Base,  the  Army’s  Defense  Supply  Serviee- 
Washington,  and  the  Navy’s  Fleet  Industrial  Supply  Center-Norfolk.  The  GAO’s  (2001) 
research  design  did  not  attempt  to  identify  a  usage  rate,  and  “data  was  not  eollected  to 
provide  a  basis  for  measuring  whether  the  test  program  produeed  the  desired  results”  (p. 
4).  Although  the  GAO  refereneed  an  Office  of  Federal  Procurement  Policy  (OFPP) 
survey  that  showed  proeurement  exeeutives’  belief  in  the  effeetiveness  of  the  program, 
empirieal  data  was  not  eolleeted,  and  the  GAO  (2001)  ultimately  eoneluded  that  the 
benefits  eould  not  yet  be  elearly  demonstrated. 

A  2005  Naval  Postgraduate  School  (NPS)  joint  applied  project  examined  the 
Marine  Corps’  Regional  Contraeting  Offiee-Southwest  (RCOSW),  loeated  at  Camp 
Pendleton,  to  determine  whether  the  offiee  was  fully  utilizing  simplified  aequisition 
legislation,  sueh  as  those  offered  in  FAR  Subpart  13.5  (Gillespie,  2005).  The  project  did 
not  use  Federal  Proeurement  Data  System  (FPDS)  data.  Instead,  it  used  data  pulled  from 
RCOSW’ s  awards  database  for  FY  2003,  2004,  and  2005.  The  study  did  not  provide  a 
FAR  Subpart  13.5  usage  rate  but  showed  that  while  90%  of  RCOSW’s  spending  was 


17 


eligible  for  simplified  aequisition  proeedures  (SAP),  sueh  as  the  test  program,  only  66% 
used  SAP.  The  study  eoneluded  that  the  remaining  aetions  were  proeured  “using 
traditional  eontraet  methods,  even  though  they  fit  the  eriteria  of  the  FAR  13.5  test 
program  and  eould  have  been  bought  using  SAP”  (Gillespie,  2005,  p.  25).  The  benefits  of 
this  study  are  that  it  utilized  a  more  data-driven  and  quantifiable  researeh  approaeh  than 
the  2001  GAO  report  and  avoided  any  FPDS  aeeuraey  issues  by  using  the  RCOSW’s 
loeal  system. 

In  2006,  an  Aequisition  Researeh  Program  (ARP)  report  by  E.  Cory  Yoder  used 
FPDS  data  to  empirieally  analyze  the  FAR  13.5  test  program  usage  rate  in  the  Navy.  The 
study  examined  the  Naval  Supply  Systems  Command  (NAVSUP)  and  the  Navy’s  Fleet 
and  Industrial  Supply  Center-San  Diego  (FISCSD)  through  both  FPDS  and  eontraet  file 
review  to  determine  the  extent  of  FAR  13.5  program  use.  The  analysis  identified  usage 
rates  from  FPDS  (as  shown  in  Table  2)  and  utilization  effeetiveness  through  eontraet 
inspeetion.  The  usage  rates  by  dollar  value  varied  signifieantly  from  13%  to  56%  while 
the  usage  rates  by  aetion  were  fairly  eonsistent  between  NAVSUP  and  FISCSD  and  were 
as  high  as  57%  in  FY  2001  (Yoder,  2006).  Despite  the  rather  positive  usage  rates, 
eontraet  inspeetion  at  FISCSD  showed  that  eontraets  eoded  as  using  the  FAR  13.5  test 
program  often  did  not  fully  utilize  all  advantages  provided  by  the  program  (Yoder,  2006). 
This  study’s  methodology  effeetively  utilized  FPDS  to  derive  reliable  usage  rates,  but 
also  reeognized  the  limitations  of  FPDS  leading  up  to  2006  and  utilized  eontraet 
inspeetion  to  provide  a  more  eomplete  analysis. 
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Table  2.  NAVSUP  and  FISCSD  FAR  13.5  Utilization  Rate 

(after  Yoder,  2006) 


ORGANIZATION 

DATA  SET  YEAR 

USAGE  RATE  ($) 

USAGE  RATE 
(ACTIONS) 

NAVSUP 

FY  2001 

36% 

45% 

NAVSUP 

FY  2002 

23% 

28% 

NAVSUP 

FY  2003 

24% 

28% 

NAVSUP 

FY  2004 

34% 

46% 

NAVSUP 

FY  2005 

27% 

28% 

Navy  FISCSD 

FY  2001 

13% 

57% 

Navy  FISCSD 

FY  2002 

15% 

20% 

Navy  FISCSD 

FY  2003 

37% 

38% 

Navy  FISCSD 

FY  2004 

56% 

53% 

Navy  FISCSD 

FY  2005 

26% 

19% 

In  2011,  a  Defense  Acquisition  University  (DAU)  study  leveraged  Yoder’s  work 
to  examine  FAR  13.5  test  progiam  utilization  at  the  Defense  Logistics  Agency  (DLA: 
Shearer,  2011).  This  research  took  advantage  of  the  improvements  to  the  Federal 
Procmement  Data  System-Next  Generation  (FPDS-NG)  data  accuracy  and  the  newly 
added  FPDS-NG  data  element  lOJ  to  derive  a  usage  rate  (by  action)  of  6%  for  FY  2009 
and  11%  for  FY  2010  (Shearer,  2011).  However,  the  data  mining  methodology  did  not 
appear  to  have  controlled  for  actions  that  were  ineligible  for  the  progiam,  such  as  limits 
by  dollar  thresholds.  Therefore,  the  usage  rates  provided  in  Table  3  appear  to  answer  the 
basic  question  of  how  often  the  FAR  13.5  test  program  is  used  in  DLA,  regardless  of 
whether  the  action  was  eligible. 


19 


Table  3.  Defense  Logistics  Agency  FAR  13.5  Utilization  Rate 

(after  Shearer,  2011) 


AGENCY 

DATA  SET  ’YEAR 

USAGE  RATE  ($) 

USAGE  RATE 
(ACTIONS) 

DLA 

FY  2009 

not  provided 

6% 

DLA 

FY  2010 

not  provided 

11% 

While  the  DAU  was  leveraging  Yoder’s  2006  study,  Yoder  and  Hawkins  (2011) 
were  developing  a  new  methodology  to  derive  the  FAR  13.5  utilization  rate  using  FPDS— 
NG  data  for  the  Ah  Force  and  Navy.  Instead  of  relying  strictly  on  the  new  FPDS-NG 
Data  Element  lOJ,  then  research  used  the  ninth  digit  of  the  solicitation  number  as  an 
alternative  indicator  (Yoder  &  Hawkins,  2011).  After  controlling  for  factors  that  would 
make  an  action  ineligible  for  the  test  program,  such  as  dollar  thresholds,  modifications, 
delivery  orders,  and  non-commerciality,  then  research  (depicted  in  Table  4)  showed  a 
78%  arrd  79%  itsage  rate  for  the  Ah  Force  arrd  Navy,  respectively  (Yoder  &  Hawkins, 
2011). 


Table  4.  Ah  Force  and  Navy  FAR  13.5  Utilization  Rate 
(after  Yoder  &  Hawkins,  2011) 


AGENCY 

DATA  SET  ’YEAR 

USAGE  RATE  ($) 

USAGE  RATE 
(ACTIONS) 

Air  Force 

FY  2010 

not  provided 

78% 

Navy 

FY  2010 

not  provided 

79% 

A  comparison  between  Yoder  and  Hawkins’s  research  and  a  recent  study  by  the 
GAO  in  2014  highlights  the  strong  effect  that  methodology  design  can  have  on  the  FAR 
13.5  test  program  irsage  rate.  The  GAO  exauhued  FY  2011  FPDS-NG  data  fiom  the 
DOD,  DOT  and  Homeland  Secirrity  and  determined  the  rrsage  rate  was  9%  by  dollar 
vahte  and  12%  by  total  actions  (GAO,  2014).  Additionally,  the  report  detenuiued  the 
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usage  rate  for  the  entiie  federal  govenmient  was  only  2%.  Complete  smiunaries  of  the 
GAO’s  findings  are  consolidated  in  Table  5.  The  GAO  report  methodology  relied 
heavily  on  the  FPDS-NG  Data  Element  lOJ  and,  unlike  Yoder  and  Hawkins,  was 
controlled  solely  by  dollar  threshold  to  remove  ineligible  or  impractical  actions. 


Table  5.  Total  Federal.  DOD,  DOI,  and  DHS  FAR  13.5  Utilization  Rate 

(after  GAO,  2014) 


AGENCY 

DATA  SET 
YEAR 

USAGE  RATE 
($) 

USAGE  RATE 
(ACTIONS) 

RESEARCH 

GROUP 

Federal 

Government 

FY2011 

2% 

Not  Provided 

GAO 

DOD,  DOI,  & 
DHS 

FY  2011 

9% 

12% 

GAO 

Army  Materiel 
Coimnand 

FY2011 

6% 

7% 

GAO 

Naval  Supply 
Systems 
Command 

FY2011 

23% 

26% 

GAO 

Air  Force 
Materiel 
Command 

FY2011 

6% 

6% 

GAO 

U.S.  Coast 
Guard 

FY  2011 

17% 

24% 

GAO 

DOI 

Acquisition 

Services 

Duectorate 

FY2011 

18% 

22% 

GAO 

The  large  variation  in  the  usage  rate  shown  by  the  last  15  year  s  of  research  is  due 
largely  to  the  variation  in  methodology  design  for  each  stirdy.  The  change  in 
methodology  over  the  years  coincides  with  the  change  in  availability  and  acciuacy  of  the 
data  set.  FPDS-NG.  As  FPDS-NG  improved  in  capabihty  and  acciuacy,  usage  rate 
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studies  applied  different  approaches  to  answer  the  FAR  13.5  usage  rate  question.  The 
next  section  discusses  the  development  of  FPDS-NG,  the  improvements  over  the  years, 
and  the  studies  conducted  regarding  FPDS-NG  accuracy. 

E.  USE  OF  THE  FEDERAL  PROCUREMENT  DATA  SYSTEM-NEXT 

GENERATION 

The  FPDS-NG  system  has  made  several  significant  improvements  over  the 
years.  Those  improvements  have  enhanced  the  ability  of  researchers  to  conduct  studies 
quantifying  the  test  program’s  usage  and  overall  benefits.  In  conducting  those  studies, 
researchers  used  various  methodology  designs  and  data  sets,  which  resulted  in  varying 
usage  rates.  This  section  explains  the  history  of  events  leading  to  the  current  state  of 
FPDS-NG  as  well  as  past  studies  to  discuss  FPDS’  accuracy  and  reliability 
improvements. 

1,  Chronological  Development  of  FPDS-NG 

The  FPDS-NG  is  an  automated  web-based  information  system  used  to  develop, 
collect,  and  disseminate  federal  procurement  data  (FPDS-NG,  n.d.).  It  is  the  only 
approved  federal  procurement  data  repository  authorized  for  use  by  Congress,  the 
Executive  Branch,  audit  agencies  such  as  the  GAO,  private  sector  companies,  and  the 
general  public  (FPDS-NG,  2014).  FPDS  was  established  under  the  direction  of  the 
OFPP,  as  amended,  41  U.S.C.  401  et  seq.  and  the  FAR  Subpart  4.6  (FPDS-NG,  2014) 
and  has  served  as  the  federal  government’s  primary  depository  for  contract  data  since 
1978  (GAO,  2003b).  This  report  uses  FPDS-NG  as  its  primary  data  source  for  evaluating 
the  usage  rate  of  FAR  Subpart  13.5,  Test  Program  for  Certain  Commercial  Items. 

The  following  time  line  provides  a  brief  history  of  FPDS-NG  and  a  summary  of 
the  events  leading  to  the  current  version  of  FPDS-NG: 

•  December  1972:  The  Commission  on  Government  Procurement 
determined  that  the  government  should  have  a  single  system  responsible 
for  collecting  and  reporting  on  government  procurement  statistics  (GAO, 
1980). 
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•  November  1973:  An  interageney  task  group  was  eharged  with  studying  the 
eommission’s  reeommendation  and  obtain  ageney  eomments  (GAO, 
1980). 

•  August  1974:  “Congress  passed  the  Offiee  of  Federal  Proeurement  Poliey 
Aet  (Publie  Law  93-400)”  (GAO,  1980,  p.  2)  requiring  the  OFPP 
administrator  to  establish  a  proeurement  traeking  system  (GAO,  1980). 

•  February  1978:  The  OFPP  administrator  designated  the  DOD  as  OFPP’s 
exeeutive  agent  in  eharge  of  establishing  and  operating  the  Federal 
Procurement  Data  Center  (FPDC)  and  FPDS  (GAO,  1980). 

•  October  1978:  Federal  agencies  were  required  to  start  collecting  identified 
data  for  FPDS  (GAO,  1980). 

•  October  1979:  Congress  enacted  Public  Law  96-83  transferring 
responsibility  of  the  FPDC  from  the  DOD  to  the  General  Services 
Administration  (GSA;  GAO,  1980). 

•  April  2003:  The  GSA  awarded  a  contract  to  Global  Computer  Enterprises 
(GCE),  Inc.,  to  review  and  transfer  federal  agencies’  data  to  EPDS-NG. 
The  contract  also  required  that  GCE  validate  and  connect  agencies 
contract  writing  systems  to  EPDS-NG  (GAO,  2005). 

•  October  2003:  EPDS-NG  became  operational  (GAO,  2003b). 

•  December  2004:  EPDS-NG  was  available  for  public  use  (GAO,  2005). 

•  September  2006:  “The  Eederal  Eunding  Accountability  and  Transparency 
Act  of  2006  (EEATA,  P.E.  109-282)  mandated  the  development  of  a  user- 
friendly  system  comprising  a  variety  of  government  spending  data, 
including  procurement  data”  (Halchin,  2013,  p.  3).  The  user-friendly 
searchable  website  developed  was  usaspending.gov. 

•  March  2007:  The  0MB  issued  a  memorandum  to  start  the  Federal 
Procurement  Data  Verification  and  Validation  (V&V). 

•  December  2007:  Usaspending.gov  was  officially  launched  (USA 
Spending,  n.d.). 

•  2009:  Data  Eield  lOJ,  “Commercial  Item  Test  Program”  was  added  to 
EPDS-NG  (GAO,  2014). 

The  most  significant  upgrade  from  EPDS  to  EPDS-NG  included  more  “machine- 
to-machine”  automated  data  entry  methods  and  error  detection  software  designed  to 
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provide  immediate  data  verification  (GAO,  2003b).  Automated  data  entry  allowed  the 
agencies’  contract  writing  systems  to  directly  transfer  contract  data  to  FPDS-NG,  and  the 
error  detection  software  enabled  the  user  to  correct  mistakes  prior  to  the  data  being 
finalized.  Both  of  these  features  were  included  in  an  effort  to  improve  the  reliability  of 
the  data  (GAO,  2003b). 

2.  FPDS-NG  FAR  Subpart  13,5  Coding  Accuracy 

Reliability  of  FPDS  data  has  been  an  issue  since  the  program’s  inception.  The 
OFPP  and  GSA  have  continually  focused  on  making  improvements  to  the  system  over 
the  years  and  have  made  great  strides  since  agencies  first  started  providing  data  in  1978. 
Currently,  OFPP  and  GSA  require  each  agency  to  complete  an  annual  certification  of 
accuracy  and  completeness  percentages  of  their  FPDS-NG  data  (FPDS-NG,  2014).  This 
section  discusses  past  FPDS-NG  data  accuracy  issues  and  the  improvements  made  to  the 
system. 

Less  than  a  year  after  agencies  started  reporting  procurement  data,  the 
Subcommittee  on  Human  Resources  and  the  Committee  on  Post  Office  and  Civil  Service 
requested  that  the  comptroller  general  examine  the  effectiveness  of  FPDS  (GAO,  1979). 
Despite  56  agencies  being  required  to  report,  20  agencies  had  either  “not  reported, 
reported  in  part,  or  submitted  reports  not  in  accordance  with  prescribed  instructions” 
GAO,  1979,  p.  1).  The  comptroller  general  reported  that 

the  Federal  Procurement  Data  System  relies  on  the  integrity  of  many 
individuals  to  prepare  reports  and  to  prepare  them  correctly.  If,  for  some 
reason,  a  report  is  not  prepared,  the  data  on  the  contract  award  will  not 
enter  the  system.  Furthermore,  the  Center  has  no  means  for  knowing 
whether  data  are  reported  for  all  contracts.  (GAO,  1979,  p.  2) 

The  comptroller  general  and  the  GAO  quickly  followed  this  1979  report  with  their 
own  FPDS  examination.  This  1979  report  was  quickly  followed  by  another  FPDS 
examination  by  the  comptroller  general  and  the  GAO.  In  April  of  1980,  they  reported 
that,  while  progress  was  made  setting  up  the  system,  “FPDS  failed  to  alert  users  that  the 
system  contained  known  errors”  (GAO,  2003b,  p.  2).  The  report  also  stated  that 
“timeliness  of  reporting  by  the  agencies  and  accuracy  of  the  data  submitted”  needed 
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improvement  (GAO,  1980,  p.  ii).  FPDS  was  in  its  infancy  and  was  already  facing  tough 
scrutiny. 

Inspections  and  reports  addressing  FPDS  accuracy  and  overall  system 
improvement  continued  to  be  a  regular  occurrence  for  the  OFPP  and  GSA  as  they  worked 
to  improve  the  system.  A  1994  GAO  report  documented  a  meeting  between  the  GAO, 
OFPP,  and  GSA  regarding  improvement  recommendations.  One  of  the  report’s  biggest 
improvement  suggestions  involved  developing  accuracy  and  completeness  standards  for 
FPDS  data.  The  OFPP  and  GSA  agreed  with  the  GAO’s  recommendations  and  tasked  the 
Federal  Procurement  Data  Center  (FPDC)  to  establish  a  set  of  standards  for  distribution 
to  all  agencies  (GAO,  1994). 

Despite  improvements  by  the  OFPP  and  GSA,  accuracy  issues  continued  to  be  a 
problem  for  the  FPDS.  For  instance,  in  a  study  to  examine  the  usefulness  of  the 
commercial  item  test  program,  the  GAO  (2001)  examined  procurement  data  and  initially 
determined  that  the  federal  government  “purchased  about  $31.6  billion  in  commercial 
items  for  fiscal  year  2000  and  used  the  simplified  procedures  to  purchase  $1.9  billion  in 
commercial  items”  (p.  3).  However,  interviews  with  the  actual  agencies  showed  that  the 
number  of  contracts  using  the  test  program  was  overstated.  The  GAO  (2001)  was  not  able 
to  evaluate  the  full  extent  of  the  possible  over-reporting  problem. 

One  of  the  most  significant  reports  to  highlight  accuracy  issues  in  FPDS  was  a 

September  2003  GAO  report  that  attempted  to  examine  the  usage  of  the  commercial  item 

test  program  for  the  DOD,  Department  of  Treasury  (DOT),  and  Department  of  Justice 

(DOJ).  The  report  found  FPDS  data  to  include  significant  errors  and  came  to  the  overall 

assessment  that  “there  is  no  reliable  information  for  measuring  the  test  program’s 

benefits”  (GAO,  2003a,  p.  7).  Figure  1  is  an  excerpt  from  GAO  report  GAO-03-1068  and 

shows  the  substantial  over  and  understatement  of  the  value  of  the  contracts  that  FPDS 

showed  as  using  the  commercial  item  test  program.  For  example,  FPDS  showed  that  the 

DoT’s  U.S.  Mint  used  the  program  for  $242  million  in  contract  actions  for  FY  2001. 

However,  through  interviews  with  DoT  procurement  officials,  the  GAO  (2003a) 

determined  that  U.S.  Mint  did  not  use  the  commercial  item  test  program  that  year. 

Conversely,  FPDS  showed  a  $4  million  value  for  the  DLA,  only  to  determine  through 
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interviews  with  DLA  proeurement  officials  that  the  value  was  actually  $146  million 
(GAO,  2003a). 


Department's 

buying 

organization 

Value  ot  test  program 
contracts,  according  to 
FPDS  (fiscai  year  2001) 

Vlfbat  piDcurement  officials 
said  about  FPOS's  data 

Department  ot  the 
Treasury’s  U.S.  Mint 

S242  miilion 

U.S.  M  nl  said  I  did  not  use  the 
test  program  at  all 

DOO'S  Defense 

Logistics  Agerx:y 

S4  million 

Oelerise  Log'st^cs  Agency  said 
a  obligated  SI  46  million  in  lest 
program  contracts 

Department  ot 

Jushce's  Federal 

Pr«on  'ndusir  es 

S1 18  million 

After  review’ng  portors  ol 
FPDS's  data,  about  S31  million 
in  contract  actions.  Federal 
Prison  Industries  said  none  ot 
those  items  were  purchased 
under  the  test  program 

Figure  1.  Examples 

of  EPDS  Discrepancies  (from  GAO,  2003a,  p.  5) 

Two  months  after  the  September  report,  the  GAO  sent  a  letter  to  the  director  of 
the  0MB  summarizing  examples  of  FPDS  data  reliability  issues  found  in  past  GAO 
studies  (GAO,  2003b).  The  letter  recognized  that  a  major  cause  of  the  errors  in  FPDS  was 
data  entry  mistakes  by  the  agencies’  contracting  personnel.  It  also  highlighted  the 
positive  attributes  of  the  new  FPDS-NG  that  addressed  data  entry  errors  and  made  the 
following  three  recommendations  for  the  improvement  of  data  reliability  in  the  new 
system; 

1 .  Ensure  that  agencies  allocate  the  resources  necessary  to  implement 
contract  writing  systems  capable  of  electronic  transfer  of  information  to 
FPDS-NG. 

2.  Require  agencies  that  have  not  yet  implemented  electronic  contract 
writing  systems  to  report  regularly  to  the  0MB  on  their  plans  to  ensure 
reliability  of  the  information  reported  to  FPDS-NG. 

3.  Request  that  major  agencies,  in  consultation  with  the  GSA,  conduct 
regular  reviews  of  their  procedures  for  collecting  and  reporting 
information  to  FPDS-  NG.  Agencies  should  conduct  such  reviews 
annually  until  a  satisfactory  level  of  reliability  is  achieved,  and 
periodically  thereafter.  (GAO,  2003b,  p.  3) 


The  0MB  and  GSA  both  agreed  with  the  recommendations,  and  on  August  25, 
2004,  the  0MB  sent  a  memo  to  the  heads  of  executive  departments  and  agencies  and  the 
President’s  Management  Council  directing  them  to  take  steps  necessary  to  comply  with 
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the  GAO’s  recommendations  (Johnson,  2004).  The  upgrade  to  FPDS-NG  and  the 
required  actions  in  this  memo  represented  a  significant  improvement  of  procurement  data 
reliability. 

In  September  2005,  the  GAO  issued  another  letter  to  the  0MB  summarizing  a 
review  of  the  FPDS-NG  improvements  to  date  (GAO,  2005).  The  report  raised  concerns 
about  federal  agencies’  lack  of  progress  toward  accuracy  and  timeliness  of  data,  which 
largely  stemmed  from  contract  writing  systems  not  yet  electronically  interfacing  with 
FPDS-NG.  While  the  report  noted  that  90%  of  agencies  made  the  connection,  it 
highlighted  the  DOD  specifically  for  its  implementation  delays  (GAO,  2005).  The  GAO 
report  recommended  that  the  0MB  work  directly  with  the  DOD  to  connect  its  contract 
writing  systems  to  FPDS-NG.  A  memorandum  from  the  director  of  Defense  Procurement 
and  Acquisition  Policy  (DPAP)  to  the  defense  agencies  dated  May  1,  2007  cited  the 
challenges  with  the  transition  to  FPDS-NG,  but  asserted  that  “the  majority  of  DOD 
contracting  offices  are  now  reporting  their  FY07  contract  actions  to  FPDS-NG”  (DPAP, 
2007,  p.  1).  While  adjustments  and  improvements  to  FPDS-NG  were  ongoing,  the 
connection  of  the  DOD’s  contract  writing  systems  was  a  significant  step  in  finalizing  the 
transition  to  the  new  system  and  overall;  2007  showed  increased  focus  and  progress 
toward  the  goal  of  data  accuracy. 

A  significant  example  of  increased  focus  and  progress  toward  data  accuracy  is 
displayed  in  the  0MB ’s  (2007)  Federal  Procurement  Data  Verification  and  Validation 
(V&V)  program  that  required  agencies  to  “[ejstablish  an  agency-wide  requirement  for 
routine,  statistically-valid  data  verification  and  validation”  (Denett,  2007,  p.  1)  and 
provide  an  annual  certification  of  their  FPDS-NG  data.  The  0MB  issued  a  memorandum 
on  May  9,  2008,  which  provided  guidance  on  improving  data  quality  and  gave  further 
instructions  on  the  new  V&V  program.  In  response  to  the  0MB ’s  V&V  program  and 
overall  focus  on  FPDS-NG  data  quality,  the  DOD  created  an  FPDS  Contract  Reporting 
Data  Improvement  Plan,  which  included  a  10-step  process  for  data  V&V  (DPAP,  2008). 
The  plan  was  sent  to  the  0MB  and  all  DOD  agencies  for  implementation  on  July  25, 
2008  (DPAP,  2008). 
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As  the  transition  to  FPDS-NG  continued  and  focus  on  data  accuracy  V&V 
increased,  the  studies  of  the  Test  Program  for  Certain  Commercial  Items  persisted.  In  an 
effort  to  improve  data  collection  on  the  use  of  the  test  program,  the  DOD  requested  an 
FPDS-NG  change  (GAO,  2014).  “FPDS-NG  was  modified  in  2009  to  include  a  specific, 
single  reporting  field  to  identify  test  program  contracts”  (GAO,  2014,  p.  9).  The  change 
resulted  in  the  addition  of  data  element  lOJ,  Commercial  Item  Test  Program  (FPDS-NG, 
2014). 

In  2009,  the  GAO  (2009)  recognized  the  improvements  of  FPDS-NG  but 
continued  to  report  data  quality  concerns  with  the  system.  A  September  2009  GAO  report 
noted  that 

submitting  data  electronically  has  improved  the  reliability  of  FPDS-NG, 
and  while  we  have  found  some  FPDS-NG  data  sufficiently  reliable  for 
specific  reports  since  our  last  review  of  the  system  in  2005,  recent  GAO 
reports  illustrate  that  the  quality  of  some  FPDS-NG  data  remains  a 
concern,  (p.  4) 

This  mixed  review  highlighted  the  progress  made  through  electronically 
submitted  contract  data,  referencing  a  GSA  report  that  stated  that  “more  than  99  percent 
of  data  in  FPDS-NG  data  were  being  submitted  to  the  system  electronically  and  that  the 
agencies  submitting  the  data  had  reviewed  and  verified  the  accuracy  and  completeness  of 
their  data”  (GAO,  2009,  p.  4).  While  the  overall  report  voiced  data  reliability  concerns 
about  several  government  systems,  including  FPDS-NG,  the  report  included  positive 
comments  about  the  data  accuracy  improvements  made  with  FPDS-NG. 

As  data  reliability  improved,  the  0MB  continued  to  push  for  accurate,  complete, 
and  timely  procurement  data  through  its  V&V  program.  In  May  2011,  the  0MB  issued 
the  memorandum  Improving  Federal  Procurement  Data  Quality:  Guidance  for  Annual 
Verification  and  Validation,  which  expanded  on  the  reporting  requirements  of  FAR  4.604 
and  looked  to  standardize  the  V&V  process  by  providing  sampling  methodologies  and 
reporting  templates  (Gordon,  2011). 

FPDS-NG  data  reliability  improved  greatly  since  its  inception  in  1978.  A  USA 
Spending  (n.d.)  report  titled  Federal  Government  Procurement  Data  Quality  Summary: 
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Fiscal  Years  2008  through  2011 — For  Agency  Data  in  the  Federal  Procurement  Data 
System  found  that  98.3%  of  contract  actions  from  2008  through  2011  were  timely  and 
94%  of  the  prescribed  data  elements  were  aecurate.  While  room  for  improvement 
remained,  clearly,  FPDS-NG  coding  accuracy  was  headed  in  the  right  direetion.  This 
positive  outlook  is  supported  by  a  2013  report  by  the  Center  for  Strategie  &  International 
Studies  (CSIS),  whieh  found  that  while  issues  remain  to  be  addressed,  “Continued  efforts 
by  both  legislative  and  user-generated  feedbaek  to  FPDS-NG  have  led  to  measurable 
ehanges  in  data  quality”  (Dadsetan  &  Raghavan,  2013,  p.  2).  It  eontinued,  “The 
combination  of  bottom-up  and  top-down  efforts  signals  a  promising  shift  in  the  quality 
and  quantity  of  future  government  contracting  data  management,  as  better  data  attracts 
larger  user-bases  and  generates  more  feedback”  (Dadsetan  &  Raghavan,  2013,  p.  2).  The 
positive  outlook  presented  in  these  statements  shows  significant  progress  made. 

Finally,  the  most  signifieant,  eurrent,  and  pertinent  report  on  FPDS-NG  data 
reliability  came  from  the  GAO  in  February  of  2014.  The  report  speeifically  assessed  the 
reliability  of  the  eommercial  item  test  program  data  in  FPDS-NG  by  randomly  sampling 
243  contracts  from  FY  2011  and  requiring  the  DOD,  DOl,  and  Homeland  Security  to 
verify  the  FPDS-NG  eoding  (GAO,  2014).  The  departments  were  then  required  to  submit 
eontraet  file  doeumentation  proving  use  of  the  eommereial  test  program.  From  the 
sample,  the  GAO  (2014)  “determined  that  the  test  program  data  reported  in  FPDS-NG 
were  sufficiently  reliable  to  determine  the  minimum  extent  to  whieh  the  test  program  was 
used  by  the  eomponents”  (p.  27).  This  study  did  not,  however,  examine  eontraets  that 
were  not  coded  as  using  the  commercial  item  test  program  and  therefore  “did  not  assess 
the  extent  to  which  test  program  actions  and  obligations  may  be  underreported”  (GAO, 
2014,  p.  27).  Therefore,  this  study  shows  that  contraeting  offieials  are  generally  not 
erroneously  eoding  purchases  using  the  commercial  item  test  program  and  overstating  the 
use,  but  it  does  not  address  the  possibility  of  understatement  through  the  laek  of  eoding 
aetual  test  programs  eontraets. 

Although  there  are  still  opportunities  to  improve  FPDS-NG,  the  latest  reports 
show  accuracy  has  increased  substantially,  particularly  with  FAR  13.5  test  program  data. 
In  addition  to  FPDS-NG  being  the  only  approved  federal  procurement  data  repository 

29 


authorized  for  use  by  Congress  sinee  1978  (GAO,  2003b),  the  reeent  improved  aecuraey 
make  it  the  most  logieal  and  sound  data  souree  available  for  studying  the  FAR  13.5  usage 
rate  in  the  Air  Foree. 

F.  CHAPTER  SUMMARY 

Previous  literature  pertaining  to  FAR  Subpart  13.5  Test  Program  for  Certain 
Commereial  Items  shows  that  a  program  meant  to  simplify  the  aequisition  proeess  has  a 
eomplex  history.  The  program’s  usage  and  overall  benefits  have  been  questioned  sinee  its 
ineeption  by  the  Federal  Aequisition  Streamlining  Aet  of  1994  and  Federal  Aequisition 
Reform  Aet  of  1996.  While  a  review  of  FAR  13.5  polieies  and  proeedures  showed  the 
program  has  a  elear  goal  of  providing  ageneies  four  fairly  straightforward  benefits  of 
simplified  aequisition  proeedures  (redueed  administrative  eosts,  improved  small  business 
opportunities,  greater  effieieneies,  and  reduced  burdens),  the  complexity  of  a  simple 
program  arises  when  groups,  such  as  GAO  and  academic  researchers,  attempt  to  quantify 
these  benefits  and  determine  overall  usage  rates.  Specifically,  early  GAO  reports 
struggled  to  find  empirical  data  to  determine  the  benefits  of  the  program.  This  was 
largely  due  to  reliability  issues  with  the  federal  governments’  contract  reporting  tool, 
FPDS-NG.  However,  as  FPDS-NG  improved  over  the  years,  the  ability  to  quantify  the 
test  program’s  usage  and  overall  benefits  improved  as  well  and  recent  studies  have  shown 
the  program’s  value.  In  conducting  those  studies,  researchers  used  various  methodology 
designs  and  data  sets,  which  resulted  in  varying  usage  rates.  Chapter  111  explains  how  this 
paper’s  methodology  builds  on  those  past  studies  and  leverages  the  improvements  of 
FPDS-NG. 
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III.  METHODOLOGY 


A,  CHAPTER  INTRODUCTION 

The  purpose  of  this  chapter  is  to  explain  the  methodology  and  limitations 
associated  with  this  research.  First,  this  chapter  will  explain  how  the  literature  review  was 
used  to  create  a  research  schema  devised  to  improve  on  past  FAR  13.5  test  program 
studies.  A  detailed  step-by-step  process,  using  the  U.S.  Air  Force  data  from  February  1, 
2013,  to  May  30,  2014,  will  be  provided  in  a  manner  that  allows  duplication  with  other 
data  sets.  Lastly,  three  limitations  associated  with  this  research  are  identified  and 
discussed. 

B,  METHODOLOGY 

The  research  methodology  for  this  project  consists  of  a  literature  review  and 
analysis  of  FPDS-NG  data  to  determine  the  usage  rate  of  the  Test  Program  for  Certain 
Commercial  Items.  The  literature  review  examined  legislature  and  federal  government 
policy  behind  the  establishment  of  the  test  program,  the  design  and  benefits  of  the 
program,  previous  usage  rate  studies,  and  accuracy  of  the  FPDS-NG.  The  analysis  of 
FPDS-NG  data  uses  a  schema  designed  to  control  for  factors  such  as  dollar  thresholds 
and  mandatory  sources  to  capture  the  most  accurate  test  program  usage  rate  for  the  U.S. 
Air  Force  from  February  1,  2013,  to  May  30,  2014.  The  precise  step-by-step  process  and 
rationale  is  outlined  in  the  following  section. 

The  data  set  contained  all  Air  Force  contract  actions  reported  in  FPDS-NG  from 
February  1,  2013,  to  May  30,  2014.  The  data  was  located  in  an  Excel  file,  and  the 
following  sorting  and  elimination  steps  were  conducted; 

Step  1:  Sort  Column  FJ  (Base  and  all  Options  Value — Data  Element  3A)  by  dollar 
amount. 

Rationale:  According  to  the  FPDS-NG  “User  Manual”  (2014), 

This  data  element  is  required  for  all  Awards  and  Modifications.  Enter  the 

mutually  agreed  upon  total  contract  or  order  value  including  all  options  (if 

any).  Eor  modifications,  this  is  the  change  (positive  or  negative,  if  any)  in 
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the  mutually  agreed  upon  total  eontract  value.  See  Data  Dietionary 
Element  3A  Use  Case  for  appropriate  data  entry  requirements,  (p.  22) 

FAR  Subpart  13.5,  Test  Program  for  Certain  Commereial  Items,  applies  to 
eontract  actions  for  the  “acquisition  of  supplies  and  services  in  amounts  greater  than  the 
simplified  acquisition  threshold  but  not  exceeding  $6.5  million  ($12  million  for 
acquisitions  as  described  in  13.500(e)),  including  options”  (FAR  Subpart  13.5,  2014). 
Sorting  by  data  element  3A  ensures  that  the  data  is  sorted  by  the  dollar  amount  that 
captures  the  base  and  all  options  instead  of  strictly  the  original  obligation  amount. 

Step  2:  Eliminate  all  actions  less  than  $150,000  and  more  than  $6.5  million. 

Rationale:  Per  FAR  Subpart  13.5,  the  Test  Program  for  Certain  Commercial  Items 
is  only  applicable  to  procurements  in  this  price  range.  Removing  all  actions  above  or 
below  those  thresholds  ensures  that  actions  that  are  ineligible  for  the  program  are 
excluded  from  the  data  set. 

Step  3:  Filter  Column  G  (Modification  Number — Data  Element  IB)  and  remove  all 
modifications. 

Rationale:  According  to  the  FPDS-NG  “User  Manual”  (2014), 

Enter  a  Modification  Number  in  accordance  with  your  agency’s  policy, 
when  reporting  modifications  to  contracts,  agreements,  or  orders.  This 
number  must  be  unique  for  any  given  PHD.  This  is  a  required  field  for  all 
modifications  and  can  be  up  to  25  characters.  Otherwise,  leave  blank,  (p. 

19) 

FAR  Subpart  13.5,  Test  Program  for  Certain  Commercial  Items,  is  not  applicable 
to  modifications.  Removing  all  actions  with  input  in  this  field  eliminates  ineligible 
modifications  from  the  data  set. 

Step  4:  Filter  Column  I  (Referenced  IDV/PIID — Data  Elements  1C,  IG,  and  IH) 
and  remove  each  action  that  has  input  in  the  cell. 

Rationale:  According  to  the  FPDS-NG  “User  Manual”  (2014), 

This  data  element  is  required  for  all  Delivery/Task  orders  and  BPA  calls. 

It  is  not  required  for  a  DCA,  Purchase  Order,  and  a  Modification.  Enter 
the  contract  or  agreement  number  of  the  IDV  against  which  your  order  is 
placed.  FPDS-NG  allows  for  up  to  50  characters.  This  data  element 
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consists  of  three  parts;  The  Referenced  IDV  agency  identifier,  Refereneed 
IDV  PHD  and  the  Referenced  IDV  Modification  Number,  (p.  19) 

This  data  element  is  only  populated  when  the  aetion  is  a  delivery  order,  task 
order,  or  blanket  purchase  agreement  (BP A)  call  against  an  existing  eontract  or 
agreement.  Writing  a  new  contract  using  the  Test  Program  for  Certain  Commereial  Items 
when  a  eontract  is  already  in  plaee  to  provide  the  supply  or  serviee  is  not  an  effieient  or 
effective  use  of  the  program.  Removing  all  actions  that  have  input  for  this  data  element 
eliminates  aetions  that  already  had  a  contract  vehicle  in  place  for  use. 


Step  5:  Filter  Column  AR  (Commercial  Item  Acquisition  Procedures — Data 
Element  lOH)  and  remove  each  action  coded  as  “Commercial  Item  Procedures  not 
used.” 

Rationale:  According  to  the  FPDS-NG  “User  Manual”  (2014), 

This  data  element  is  required  for  a  DCA,  Purehase  Order,  and  populates 
on  all  orders  and  PBA  Calls  and  Modifieations.  This  designates  whether 
the  solicitation  used  the  special  requirements  for  the  aequisition  of 
eommereial  items  or  other  supplies  or  services  authorized  to  use 
commereial  item  proeedures  intended  to  more  closely  resemble  those 
customarily  used  in  the  commereial  marketplace  as  defined  by  FAR  Part 
12.  Seleet  the  appropriate  value  from  the  drop  down  menu.  See  Data 
Dietionary  Element  lOH  Use  Case  for  appropriate  data  entry  requirements. 

(p.  55) 

FAR  Subpart  13.5,  Test  Program  for  Certain  Commereial  Items,  is  only 
applieable  to  commercial  items.  Removing  all  aetions  eoded  as  “Commereial  Item 
Procedures  not  used”  removes  all  actions  for  non-commercial  items  that  are  not  eligible 
for  the  test  program. 


Step  6:  Filter  Column  AT  (Commercial  Item  Test  Program — Data  Element  lOJ) 
and  remove  cells  with  “No,” 

Rationale:  According  to  the  FPDS-NG  “User  Manual”  (2014), 

This  data  element  is  required  for  a  DCA,  Purchase  Order,  and  populates 
on  all  orders  and  PBA  Calls  and  Modifications.  This  field  designates 
whether  the  aequisition  utilized  FAR  13.5  Test  Program  for  Certain 
Commereial  Items.  The  FAR  13.5  Test  Program  provides  for  the  use  of 
simplified  aequisition  proeedures  for  the  acquisition  of  supplies  or 
services  in  amounts  greater  than  the  simplified  acquisition  threshold  when; 
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(1)  the  acquisition  does  not  exceed  the  threshold  and  the  contracting 
officer  reasonably  expects  that  offers  will  only  include  commercial  items; 

(2)  the  acquisition  does  not  exceed  the  threshold  and  is  for  commercial 
items  that,  as  determined  by  the  head  of  the  agency,  are  to  be  used  in 
support  of  a  contingency  operation  or  to  facilitate  the  defense  against  or 
recovery  from  nuclear,  biological,  chemical,  or  radiological  attack;  or  (3) 
the  acquisition  does  not  exceed  the  threshold  and  can  be  treated  as  an 
acquisition  of  commercial  items  in  accordance  with  FAR  12.102(f)(1) — 
agency  head  determines  the  supplies  or  services  are  to  be  used  to  facilitate 
defense  against  or  recovery  from  nuclear,  biological,  chemical,  or 
radiological  attack,  (p.  56) 

This  final  sort  shows  the  total  actions  eligible  to  use  FAR  Subpart  13.5,  Test 
Program  for  Certain  Commercial  Items,  versus  how  many  actions  actually  used  the 
program. 

C.  METHODOLOGY  LIMITATIONS 

Three  main  limitations  were  identified  during  the  course  this  research.  The  first  is 
an  inherent  limitation  due  to  the  data  relying  on  human  data  entry  and  self-reporting.  As 
discussed  in  the  literature  review,  accuracy  and  reliability  issues  have  been  found  in 
FPDS-NG  reporting,  although  actions  coded  as  having  utilized  the  test  program  were 
accurate  (GAO,  2014). 

The  second  limitation  was  discovered  through  performing  data  analysis  and  stems 
from  a  discrepancy  between  two  data  fields.  Data  cell  Column  AJ  (Extent  Competed — 
Data  Element  lOA)  “is  the  code  that  represents  the  competitive  nature  of  the  contract” 
(EPDS-NG,  2014,  p.  47).  Table  6  illustrates  the  various  options  available: 
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Table  6.  Options  for  FPDS-NG  Data  Element  lOA — Extent  Competed 

(from  EPDS-NG,  2014,  p.  47) 


Short  Description 

Full  Description 

Competed  under  SAP 

Select  this  code  when  the  action  is 
competed  under  the  Simplified 

Acquisition  Threshold. 

Pollow  On  to  Competed  Action.  This 
value  is  not  available  for  Post  CEOD 
actions. 

Select  this  code  when  the  action  is  a 
follow  on  to  an  existing  competed 
contract.  PAR  6.302-1 

Pull  and  Open  Competition 

Report  this  code  if  the  action  resulted 
from  an  award  pursuant  to  PAR  6. 102(a)  - 
sealed  bid,  PAR  6.102(b)  -  competitive 
proposal,  PAR  6.102(c)  -  combination,  or 
any  other  competitive  method  that  did  not 
exclude  sources  of  any  type 

Pull  and  Open  Competition  after  exclusion 
of  sources 

Select  this  code  when  some  sources  are 
excluded  before  competition 

Not  Available  for  Competition 

Select  this  code  when  the  contract  is  not 
available  for  competition 

Not  Competed 

Select  this  code  when  the  contract  is  not 
competed. 

Not  Competed  under  SAP 

Select  this  code  when  the  action  is  NOT 
competed  under  the  Simplified 

Acquisition  Threshold 

Eor  actions  coded  as  utilizing  the  PAR  13.5  Test  Program  under  Data  Element 
lOJ,  all  actions  were  rightfully  coded  as  either  “Competed  under  SAP”  or  “Not  Competed 
under  SAP;”  however,  there  were  also  actions  in  excess  of  $150,000,  but  less  than 
$6,500,000  coded  as  “Competed  under  SAP”  or  “Not  Competed  under  SAP”  that  were 
not  coded  as  utilizing  the  PAR  13.5  Test  Program.  Generally,  this  situation  should  not 
exist  because  the  authority  for  utilizing  SAP  in  excess  of  $150,000  is  limited  to  the  test 
program,  which  authorizes  the  use  up  to  $6,500,000,  or  declared  contingencies,  which 
authorize  the  use  up  to  $300,000  inside  the  United  States  and  up  to  $1,000,000  outside 
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the  United  States  (FAR  2.101,  2014).  Of  those  aetions  that  were  eoded  as  either 
“Competed  under  SAP”  or  “Not  Competed  under  SAP,”  349  actions  were  less  than  or 
equal  to  $1,000,000,  leaving  a  potential  for  SAP  usage  in  support  of  a  contingency,  with 
49  additional  actions  that  appear  to  at  least  be  miscoded  in  one  way  or  another. 
Unfortunately,  FPDS-NG  data  does  not  contain  a  data  field  to  distinguish  between 
actions  in  support  of  a  contingency  and  those  that  are  not,  which  is  the  third  and  final 
limitation. 

The  inability  to  identify  contingency  contracting  actions  limits  the  accuracy  of  the 
methodology  in  two  ways.  First,  since  the  SAT  is  higher  for  declared  contingencies,  not 
all  actions  in  excess  of  $150,000  require  the  authority  of  FAR  13.5  to  use  SAP.  This 
could  result  in  the  methodology  somewhat  overstating  the  test  program’s  full  usage  rate; 
however,  the  overseas  contingency  contracts  issued  in  Afghanistan  use  an  Army 
Department  of  Defense  Activity  Address  Code  (DODAAC)  because  the  Army  is  the  lead 
agency  in  the  theater  of  operations.  Since  those  contracts  would  not  appear  in  an  Air 
Force  data  set  pulled  from  FPDS-NG,  the  risk  of  overstating  the  program’s  usage  rate  is 
minimal.  Second,  the  FAR  13.5  Test  Program’s  upper  threshold  for  contingencies  is 
$12,000,000.  As  a  result,  this  methodology  may  understate  the  full  usage  rate  since  any 
actions  in  support  of  a  contingency  in  excess  of  $6,500,000  were  not  counted.  Of  all 
contracts  between  $6,500,000  and  $12,000,000,  only  three  actions  were  coded  as  using 
the  test  program.  While  FPDS-NG  does  contain  a  mandatory  data  field  to  indicate  if  a 
contract  action  “supports  a  declared  contingency  operation,  a  declared  humanitarian 
operation,  or  a  declared  peacekeeping  operation”  (Contingency,  Humanitarian,  or 
Peacekeeping  Operation — Data  Element  6H),  that  field  is  not  part  of  the  FPDS-NG 
competition  report  (FPDS-NG,  2014,  p.  36).  If  that  field  was  present  in  the  data,  both  of 
the  previous  limitations  could  be  corrected  for  through  additional  filters  and  sorting  to 
specifically  identify  and  factor  in  those  additional  test  program  actions. 

Of  these  limitations,  using  an  alternate  final  step  in  the  methodology  can  assess 
the  second  issue  concerning  the  discrepancy  in  coding: 
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Alternate  Step  6:  Filter  Column  AJ  (Extent  Competed — Data  Element  lOA)  and 
keep  cells  with  “Competed  Under  SAP”  and  “Not  Competed  Under  SAP.” 

This  alternative  final  sort  shows  the  total  number  of  aetions  over  $150,000  but 
less  than  or  equal  to  $6,500,000  where  Simplified  Aequisition  Proeedures  were  used 
either  competitively  or  noncompetitively  out  of  the  same  pool  of  eligible  contracts. 
Theoretically,  this  number  should  be  identical  to  the  number  of  actions  coded  as  using  the 
test  program  since  FAR  13.5  is  the  authority  for  utilizing  SAP  in  excess  of  the  SAT.  Any 
inconsistencies  between  the  two  computed  values  would  signify  miscoding  in  one  of  the 
two  fields. 

D.  CHAPTER  SUMMARY 

The  research  methodology  described  in  this  section  leverages  the  lessons  obtained 
from  the  literature  review  to  create  a  schema  devised  to  improve  on  past  FAR  13.5  test 
program  studies.  The  result  is  a  detailed  step-by-step  process  designed  to  control  for  key 
factors,  such  as  dollar  thresholds  and  mandatory  sources,  and  ultimately  aims  to  capture 
the  most  accurate  test  program  usage  rate  possible  for  the  U.S.  Air  Force  from  February 
1,2013,  to  May  30,  2014. 

The  three  identified  limitations,  human  data  entry,  FPDS-NG  data  field 
discrepancy,  and  contingency  contracting  actions,  each  provide  a  unique  challenge. 
However,  the  design  schema  was  developed  to  reduce  the  effects  of  the  limitations  as 
much  as  possible.  Human  data  entry  and  self-reporting  have  and  will  continue  to  affect 
accuracy  levels  of  FPDS-NG  data.  However,  recent  studies  have  shown  accuracy  of 
FPDS-NG  data  pertaining  to  the  FAR  13.5  test  program  have  improved  significantly  and 
are  “sufficiently  reliable  to  determine  the  minimum  extent  to  which  the  test  program  was 
used  by  the  components”  (GAO,  2014,  p.  27).  Additionally,  despite  the  improvements  of 
the  FPDS-NG  contract  reporting  system,  errors  in  the  data  fields  still  exist.  To  account 
for  a  data  field  error  that  specifically  affects  the  FAR  13.5  test  program  data,  an  alternate 
step  in  the  methodology  was  developed  resulting  in  a  usage  range.  Finally,  while  the  data 
set  does  not  enable  the  ability  to  adjust  for  contingency  contract  actions,  avoiding  U.S. 
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Army  DODAACs  significantly  reduces  the  risk  of  overstating  the  program’s  usage  rate. 
Chapter  IV  presents  the  data  resulting  from  the  applied  methodology  and  analyzes  the 
results. 
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IV.  DATA  PRESENTATION  AND  ANALYSIS 


A,  CHAPTER  INTRODUCTION 

The  purpose  of  this  chapter  is  to  analyze  the  Air  Force’s  usage  rate  of  the  FAR 
Subpart  13.5  Test  Program  for  Certain  Commercial  Items.  First,  test  program  awards  will 
be  identified  using  the  methodology  described  in  Chapter  III  and  the  dollar  value 
distribution  examined.  Level  of  competition  and  average  degree  of  competition  will  then 
be  provided  and  analyzed.  Lastly,  Air  Force  policy,  guidance,  and  training  will  be 
discussed  as  possible  conditions  affecting  the  data  results. 

B,  ANALYZING  THE  AIR  FORCE’S  USAGE  RATE  OF  THE  TEST 

PROGRAM 

Utilizing  the  previously  defined  methodology  in  Chapter  III  produced  the 
following  results. 

1.  Identifying  Test  Program  Awards  in  the  Air  Force 

The  FPDS-NG  data  from  February  1,  2013,  through  May  31,  2014,  contained  a 
total  of  156,248  contract  actions.  These  contract  actions  consisted  of  any  new  contract 
awards,  modifications,  purchase  and  delivery  orders,  etc.,  that  were  reported  through 
FPDS-NG  during  that  period  of  time.  Of  all  those  contract  actions,  18,995  were  of  a 
dollar  value  greater  than  the  SAT  ($150,000)  but  less  than  or  equal  to  $6,500,000.  Since 
those  actions  still  contained  modifications,  factoring  those  out  resulted  in  a  total  of 
11,570  new  contracts  or  pure hase/de livery  orders  awarded  within  the  test  program’s 
range.  Next,  eliminating  new  actions  utilizing  ordering  vehicles  such  as  BP  As  and  ID/IQs 
reduced  the  number  of  actions  to  3,508.  As  non-commercial  acquisitions  were  removed, 
the  total  number  of  actions  eligible  for  the  test  program  within  the  data  set  was  1,709. 
Finally,  filtering  the  remaining  eligible  contract  actions  returned  only  475  actions 
reporting  use  of  the  test  program,  a  usage  rate  of  just  27.79%.  Use  of  the  alternate 
methodology,  which  filtered  results  based  on  the  competition  coding  instead  of  the  test 
program  coding,  on  the  same  eligible  pool  of  1,709  contract  actions  returned  880  actions, 
a  usage  rate  of  51.49%,  that  reported  the  use  of  SAP  (either  competitively  or  non- 
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competitively)  in  exeess  of  $150,000,  but  less  than  or  equal  to  $6,500,000.  Figure  2 
graphieally  depiets  the  results  of  these  filters. 


Step  1: 156,248  Actions 

Step  2: 18,995  Actions 

Step  3: 11,570  Actions 

Step  4:  3,508  Actions 

Step  5: 1,709 
Actions 

Step  6: 

475  to  880 
Actions 

Identifying  the  Air  Foree’s  Number  of  Test  Program  Awards 
between  $150,000  and  $6.5M  (after  FPDS-NG,  n.d.) 

Test  Program  for  Certain  Commereial  Items  authorizes  the  use  of 
simplified  proeedures  in  exeess  of  the  SAT,  both  methodologies  should  theoretieally 
report  the  same  result;  however,  only  475  actions  were  eoded  as  having  utilized  the  test 
program  while  880  actions  (comprised  of  those  same  475  plus  another  405)  were  eoded 
as  having  utilized  simplified  proeedures.  Beeause  of  the  ineonsistency  in  data  fields  that 
should  be  synonymous  with  one  another,  it  would  be  presumptuous  to  eite  one  result  as 
representing  the  true  usage  rate  of  the  test  program  for  the  Air  Force  while  ignoring  the 
other  or  to  assume  how  aetions  were  miseoded  without  aeeess  to  additional  eontract 
doeumentation.  Instead,  the  two  data  points  ,  27.79%  and  51.49%,  (as  shown  in  Table  7) 
should  represent  the  floor  and  eeiling,  respeetively,  for  the  use  of  test  program  provisions 
in  Air  Foree. 


Figure  2. 


Sinee  the 
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Table  7.  Air  Force  Test  Program  Awards  Between  $150,000  and  $6.5M 

(after  FPDS-NG,  n.d.) 


METHODOLOGY 

TEST  PROGRAM 
NUMBER 

TEST  PROGRAM 
PERCENT 

Primary;  Corrmiercial  Item 

Test  Program  -  Data 

475 

27.79% 

Element  lOJ 

Alteniate;  Extent  Competed 

-  Data  Element  lOA 

880 

51.49% 

Tlieoretically,  since  the  FAR  requires  contracting  activities  to  “employ  the 
simplified  procedmes  authorized  by  the  test  to  the  maximum  extent  practicable,”  the 
number  of  actions  coded  as  utilizing  the  Test  Program  for  Certain  Commercial  Items 
should  ideally  be  at  or  near  the  number  of  eligible  actions  of  1,709  (FAR  13.500(b)); 
however,  in  the  case  of  both  methodologies,  a  substantial  number  of  new  contract  actions 
failed  to  use  the  test  program.  Smce  IDV’s  were  factored  out  of  the  eligible  pool  rmder 
this  methodology,  it  stands  to  reason  that  any  non-test  program  actions  had  to  utilize  the 
more  complex,  exhaustive  procedmes  requhed  by  FAR  Pari  14,  Sealed  Biddmg,  or  FAR 
Part  15,  Contracting  by  Negotiation.  Considering  that  the  GAO  (2014)  reported  “the  test 
program  contracts  [they]  reviewed  were  generally  awarded  more  quickly  and  with  less 
administrative  biuden  than  had  the  contracts  been  awarded  rrsing  negotiated  procedmes,” 
(p.  14)  Air  Force  contracting  officials  are  seemingly  missing  oirt  on  opportimities  to 
redrrce  the  time  and  resomces  spent  on  at  least  48.51%  and  rrp  to  72.21%  of  commercial 
reqrruemerrts  (GAO,  2014). 
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2. 


Distribution  of  Air  Force  Test  Program  Awards 


Another  area  that  was  analyzed  was  the  dishibution  of  test  progr  am  usage  based 
on  the  dollar  amoimt  of  the  requirement.  Using  the  original  methodology’s  Commercial 
Item  Test  Program — Data  Element  lOJ,  dollar  values  were  separated  into  two  categories, 
test  program  and  non-test  program  actions,  then  segmented  in  $500,000  increments. 
Figiue  3  graphically  portrays  the  results. 
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Dollar  Value 


■  Test  Program  ■  Non-Test  Program 


Figiue  3.  Test  Progr  am  and  Non-Test  Progr  am  Actions  by  Dollar'  Value 

(after  FPDS-NG,  n.d.) 


Overall,  there  were  only  minor  deviations  fi'om  the  overall  average  usage  rate  of 

27.79%  with  only  two  sets  of  values  exceeding  a  difference  of  6%.  Those  two  deviations 

were  acquisitions  between  $3,500,000  and  $4,000,000,  which  totaled  48  observations 
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with  a  usage  rate  of  only  16.67%  and  aequisitions  between  $5,500,000  and  $6,000,000, 
whieh  totaled  only  10  observations  with  a  usage  rate  of  50.00%.  In  general,  this  data 
appears  to  indieate  that  any  pereeived  risk  assoeiated  with  higher  dollar  value 
aequisitions  does  not  signifieantly  impaet  the  rate  at  whieh  the  test  program  is  used  by 
Air  Foree  eontraeting  offieials. 

C.  TEST  PROGRAM  COMPETITION  STATISTICS 

In  addition  to  eontaining  data  that  provided  the  usage  rate  of  the  Test  Program  for 
Certain  Commereial  Items,  the  FPDS-NG  data  ineluded  additional  information  that  eould 
potentially  illustrate  the  effeetiveness  of  the  program.  One  foundational  eomponent  of  the 
Federal  Aequisition  System  is  eompetition.  Required  by  CICA  (unless  otherwise 
exempted),  eompetition  is  integral  toward  ensuring  effieieney,  effeetiveness,  and 
transpareney  in  government  eontraets  (Manuel,  2011).  The  GAO  reaffirms  this  notion  in 
their  report  on  eompetition  in  federal  eontraeting  by  stating  “The  benefits  of  eompetition 
in  aequiring  goods  and  serviees  from  the  private  seetor  are  well  established.  Competitive 
eontraets  ean  help  save  the  taxpayer  money,  improve  eontraetor  performanee,  eurb  fraud, 
and  promote  aeeountability  for  results”  (GAO,  2010,  p.  1).  Based  on  the  important  role  of 
eompetition  in  eontraeting,  it  simply  made  sense  to  use  the  available  data  to  assess  the 
test  program’s  ability  to  fulfill  this  publie  poliey  objeetive  and  eompare  the  results  to 
non-test  program  and  IDV  notions. 

1,  Level  of  Competition 

The  same  FPDS-NG  report  oontained  data  on  whether  an  notion  was  oompetitive 
and  even  the  amount  of  eompetition  that  was  obtained.  Column  AJ  of  the  exoel 
spreadsheet  oontained  the  “Extent  Competed — Data  Element  10 A”  and  oolumn  AS 
oontained  the  “Number  of  Offers  Reoeived — Data  Element  lOD”  (EPDS-NG,  2014). 
Multiple  filters  were  used  to  separate  test  program,  non-test  program,  and  IDV  notions. 
Next,  the  eompetition  information  from  both  data  elements  was  used  to  oaloulate  how 
many  notions  were  oompeted,  how  many  notions  were  oompeted  with  only  one  offer 
reoeived,  and  how  many  notions  were  not  oompeted.  Eigure  4  depiots  the  oaloulations  and 
oorresponding  results. 
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Figiu  e  4.  Percentage  of  Competitive,  Noncompetitive,  and  Competed 
Contracts  Actions  with  Only  One  Offer  Received 
(after  FPDS-NG,  n.d.) 

The  tigme  shows  that  test  progr  am  actions  had  the  highest  rate  of  competition  at 
75.79%  while  also  minimizing  the  instances  in  which  only  one  offer  is  received  at 
14.44%.  On  the  other  hand,  non-test  program  actions  for  commercial  items  over  the  same 
dollar  range  had  the  lowest  rate  of  competition  at  53.16%.  Meanwhile,  IDVs,  an 
increasingly  popular  acquisition  teclmique,  had  the  highest  incident  of  competition  with 
only  one  offeror  at  20.13%  (it  is  worth  noting  that  this  methodology  controlled  for 
mrrltiple  award  IDVs  to  detemrine  this  rate). 
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2.  Average  Degree  of  Competition 

Coliumi  AS,  “Number  of  Offers  Received  —  Data  Element  lOD,”  was  also  used  to 
calculate  the  average  number  of  offerors  participating  in  response  to  the  various 
acquisition  methods.  Again,  the  same  filters  were  used  to  separate  test  program,  non-test 
program,  and  IDV  actions  from  one  another  before  calculating  the  average.  Figiue  5 
displays  those  results  for  comparison. 
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Figme  5.  Average  Number  of  Offers  Received  (after  FPDS-NG,  n.d.) 


According  to  the  FPDS-NG  data  reports,  more  offerors  generally  responded  to 
requirements  solicited  using  test  progiam  procediues  than  either  non-test  program  or  IDV 
procediues.  While  the  raw  numerical  differences  between  the  various  methods  appear 
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inconsequential,  the  eomputed  numbers  are  small  to  begin  with  so  the  difference  is  best 
represented  as  a  percentage.  On  average,  for  those  actions  that  were  competed,  test 
program  solieitations  reeeived  5.60%  more  offerors  than  non- test  program  solicitations 
and  11.03%  more  offerors  than  those  utilizing  IDVs  (the  number  of  observations  for  test 
program,  non-test  program,  and  IDV  actions  were  475,  1,234,  and  2,325,  respectively). 

3,  Section  Summary 

Ultimately,  since  eorrelation  does  not  prove  causation,  these  results  do  not 
neeessarily  indicate  that  more  aetions  are  competed  or  more  offers  are  reeeived  as  a  result 
of  utilizing  test  program  proeedures;  however,  the  data  does  at  least  affirm  that  the  use  of 
the  test  program’s  simplified  and  streamlined  proeedures  has  not  eireumvented 
competition  requirements.  Furthermore,  the  signifieant  eorrelation  of  inereased 
eompetition  and  test  program  usage  seems  to  warrant  an  investigation  into  the  causal 
relationship  between  aequisition  methods  and  levels  of  competition. 

D,  TEST  PROGRAM  POLICY,  GUIDANCE,  AND  TRAINING 

In  an  effort  to  identify  what  factors  may  be  limiting  the  test  program’s  usage,  this 
paper  will  examine  the  related  policy,  guidance,  and  training  to  determine  whether  they 
have  a  positive,  negative,  or  neutral  impact. 

1.  Policy 

The  Test  Program  for  Certain  Commercial  Items  has  been  in  existence  for 
approximately  18  years  sinee  its  original  passage  under  FARA;  however,  during  that 
time,  the  program  has  always  remained  temporary,  requiring  reauthorization  by  Congress 
eaeh  time  it  expired.  The  result  is  inereased  unpredietability  for  contraeting  officials  that 
is  further  complieated  by  periods  where  the  test  program  has  outright  expired  without 
being  renewed,  like  on  January  1,  2012.  In  that  instance,  the  authority  for  the  program 
lapsed  until  over  a  year  later,  on  January  15,  2013,  when  the  NDAA  for  FY  2013  was 
passed  to  temporarily  extend  the  authority  for  the  test  program  until  January  1,  2015 
(DPAP,  2013).  The  memorandum  used  to  inform  the  DOD  aequisition  workforee  of  the 
program’s  extension  further  illustrates  the  volatile  nature  of  the  program  when  it  stated. 
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“This  class  deviation  is  effective  until  it  is  ineorporated  in  the  FAR  or  is  otherwise 
reseinded”  (DPAP,  2013,  p.  1).  In  other  words,  eontraeting  professionals  interested  in 
using  the  test  program  during  the  time  period  while  the  FAR  was  not  updated  needed  to 
further  verify  that  another  memorandum  was  not  published  reseinding  the  authority  of  the 
first  memorandum,  whieh  authorized  the  temporary  deviation.  Consider  another  situation 
in  whieh  a  eontraeting  professional  referenees  an  older  (pre-2012)  hard  eopy  of  the  FAR 
and  mistakenly  determines  that  the  test  program  is  no  longer  valid  beeause  it  lists  the 
January  1,  2012  expiration  date.  Ultimately,  the  tenuous  situation  does  not  help  the  usage 
rate  of  the  test  program,  nor  does  it  inspire  eonfidenee  for  eontraeting  offieials  to  devote 
time,  effort,  and  resourees  toward  developing  guidanee  or  training  on  the  use  of  test 
program  proeedures  and  how  to  best  take  advantage  of  them. 

2.  Guidance 

Guidanee  on  test  program  proeedures  is  searee.  Other  than  the  information 
eontained  in  FAR  Subpart  13.5,  the  DFARS  and  AFFARS  are  eompletely  silent  when  it 
eomes  to  the  test  program.  Furthermore,  the  Air  Foree  Installation  Contraeting  Ageney 
(AFICA)  Mandatory  Proeedures  (MP),  whieh  governs  operational  eontraeting  squadrons 
(the  ageneies  in  the  Air  Foree  most  likely  to  use  the  test  program  beeause  they  aequire 
primarily  eommereial  items),  also  makes  no  mention  of  the  test  program.  Therefore, 
beyond  the  requirement  at  FAR  13.500(b)  to  “employ  the  simplified  proeedures 
authorized  by  the  test  to  the  maximum  extent  praetieable,”  the  DoD  and  AF  do  nothing 
further  to  eneourage  or  require  its  use. 

3,  Training 

From  a  training  perspeetive,  the  mission  of  the  Defense  Aequisition  University 
(DAU)  is  to  “Provide  a  global  learning  environment  to  develop  qualified  aequisition, 
requirements  and  eontingeney  professionals  who  deliver  and  sustain  effeetive  and 
affordable  warfighting  eapabilities”  (DAU,  n.d.a,  para.  1).  The  DAU  is  the  primary 
organization  for  DOD  ageneies,  ineluding  the  Air  Foree,  to  obtain  eontraeting  related 
training  and  information.  Per  ieatalog.dau.mil,  the  DAU’s  offerings  are  broken  down  into 
two  eategories:  training  eourses  and  Continuous  Learning  Modules  (CLM).  The  DAU 
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currently  offers  23  different  eontraeting  eourses  with  one  eourse,  CON  237,  addressing 
Simplified  Aequisition  Proeedures  (DAU,  n.d.e).  Figure  6  displays  DAU’s  learning 
objectives  for  CON  237. 


DEFENSE  ACQUISITION  UNIVERSITY 

CON  237  -  Simplied  Acquisition  Procedures 

090210 


Course  Learning/Performance  Objectives  followed  by  its 
enabling  learning  objectives  on  separate  lines  if  specified. 


1 

Given  a  description  of  a  requirement,  identify  the  benefits  of  using  Simplified  Acquisition  Procedures  (SAP). 

Recognize  the  benefits  of  using  SAP, 

Identify  the  reguiatory  and  iegai  basis  for  SAP. 

Define  the  Simpiified  Acquisition  Threshoid  (SAT). 

2 

Given  a  description  of  a  requirement,  determine  if  Simplified  Acquisition  Procedures  (SAP)  can  be  used. 

Determine  if  a  requirement  can  be  satisfied  using  simpiified  acquisition  procedures. 

Determine  if  a  requirement  can  be  satisfied  by  purchasing  a  commerciai  or  non-deveiopmentai  item. 

Describe  the  advantages  of  purchasing  commerciai  items. 

3 

Given  a  description  of  a  requirement,  identify  potential  market  research  resources  and  techniques. 

Describe  the  reasons  to  conduct  market  research. 

Identify  market  research  techniques. 

Identify  market  research  resources. 

4 

Given  a  description  of  a  requirement,  determine  when  required  sources  and  mandatory  items  must  be  used. 

Determine  when  required  sources  and  mandatory  items  must  be  used. 

identify  the  requirements  for  smaii  business  set-asides  when  using  SAP. 

Describe  the  information  tracked  from  SAP  source  iists. 

5 

Given  a  description  of  a  requirement,  select  a  purchase  method  to  be  used  in  conjunction  with  the  Simplified  Acquisition  Procedures  (SAP). 

Determine  when  to  use  the  Governmentwide  commercial  purchase  card. 

Determine  when  it  is  practicable  to  use  FACNET/EDI. 

Describe  the  advantages  of  using  blanket  purchase  agreements. 

Identify  situations  when  written  purchase  orders  are  required. 

Determine  when  to  use  other  payment  methods  (SF  44/imprest  fund/third  party  draft). 

6 

Given  a  description  of  a  requirement,  select  a  solicitation  method  to  be  used  in  conjunction  with  the  Simplified  Acquisition  Procedures  (SAP). 

Select  the  solicitation  method  to  be  used  based  on  the  requirement. 

Identify  the  public  announcement  requirements  associated  with  each  solicitation  method. 

Determine  synopsis  requirements  associated  with  each  solicitation  method. 

Describe  the  competition  requirements  associated  with  each  solicitation  method. 

7 

Given  a  description  of  a  requirement,  determine  the  types  of  evaluation  factors  to  use  in  Simplified  Acquisition  Procedures  (SAP)  solicitations. 

Select  a  quote  based  on  price-related  factors. 

Determine  when  to  use  non-price-related  factors. 

Select  the  types  of  non-price-related  factors  to  include  in  a  solicitation  based  on  requirements  description. 

8 

Given  a  description  of  a  post-award  issue,  identify  the  actions  that  should  be  taken  according  to  FAR  Part  13. 

Describe  the  quality  assurance  limitations  when  purchasing  commercial  items. 

Describe  the  importance  of  warranties  in  purchasing  commercial  items. 

identify  when  purchase  orders  should  be  terminated  or  canceled. 

Determine  when  options  can  be  exercised  in  SAP. 

9 

Given  a  statement  describing  the  major  points  covered  in  the  preceding  lessons  on  Simplified  Acquisition  Procedures,  select  the  corresponding 
definition,  policy,  regulation,  threshold,  or  procedure. 

Page  1  of  1 

Figure  6.  DAU  Learning  Objeetives  for  CON  237 — Simplified  Aequisition 

Proeedures  (from  DAU,  n.d.a) 

Despite  an  entire  eourse  devoted  to  FAR  Part  13  and  Simplified  Aequisition 
Proeedures,  none  of  the  eourse’s  learning/performance  objeetives  make  any  mention  of 
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the  Test  Program  for  Certain  Commercial  Items.  Likewise,  the  brief  description  of  the 
CLM  covering  Simplified  Acquisition  Procedures,  CLC  005,  does  not  reference  the  test 
program  either  when  it  merely  states  “This  Simplified  Acquisition  Procedures  Overview 
(SAP)  Continuous  Learning  Module  (CLM)  is  designed  to  provide  Federal  procurement 
professionals  with  a  better  understanding  of  contracting  for  suppliers  and  services  using 
SAP”  (DAU,  2014,  para.  1). 

4,  Section  Summary 

Ultimately,  these  three  conditions  do  not  create  an  environment  conducive  to  the 
use  of  test  program  procedures.  The  lack  of  a  consistent  policy  means  the  authority  for 
the  program  is  unpredictable,  which  could  increase  buyers’  apprehension  about 
employing  the  procedures  and  negatively  impact  the  test  program’s  usage.  Furthermore, 
the  temporary  nature  of  the  test  program  is  quite  possibly  a  driving  or  contributing  factor 
to  the  vast  lack  of  training  and  educational  materials  relating  to  its  use,  as  agencies  may 
be  reluctant  to  invest  resources  in  developing  curricula  and  informative  materials  for  a 
process  that  may  not  exist  in  a  few  years.  These  lack  of  materials  mean  individuals  are 
not  exposed  to  the  existence  of  the  test  program,  how  it  functions,  or  its  merits,  which 
would  further  limit  its  usage.  Taken  together,  these  circumstances  limit  the  contracting 
professionals  in  the  Air  Force  from  employing  simplified  procedures  in  excess  of  the 
SAT,  resulting  in  a  usage  rate  that  is  less  than  optimal. 

E,  CHAPTER  SUMMARY 

This  chapter  analyzed  the  Air  Force’s  usage  rate  of  the  FAR  Subpart  13.5  Test 
Program  for  Certain  Commercial  Items  through  filtering  data  in  the  FPDS-NG 
Competition  Report.  The  chapter  also  identified  the  various  competition  metrics  and 
compared  them  to  alternate  acquisition  procedures.  Lastly,  areas  where  policy,  guidance, 
and  training  are  lacking  were  discussed  as  factors  potentially  limiting  the  use  of  test 
program  procedures. 
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V.  CONCLUSIONS  AND  RECOMMENDATIONS 


A,  CHAPTER  INTRODUCTION 

The  purpose  of  this  chapter  is  to  summarize  the  findings  of  this  research  paper. 
The  conclusion  will  summarize  the  highlights  of  the  test  program,  answer  the  research 
paper’s  two  research  questions,  then  provide  recommendations  to  support  further 
improvements  in  the  usage  of  the  test  program.  This  chapter  will  also  highlight  a  few 
areas  for  expanding  research  on  the  test  program. 

B,  CONCLUSIONS 

As  explained  in  detail  in  Chapter  II,  Section  B,  the  test  program  procedures 
authorized  by  FAR  Subpart  13.5  are  designed  to  provide  a  number  of  benefits  to 
acquisition  professionals  contracting  for  commercial  items  between  $150,000  and 
$6,500,000.  Those  benefits  are: 

1 .  Reduced  administrative  costs  by  limiting  file  documentation  and 
authorizing  a  more  efficient  acquisition  process. 

2.  Improved  opportunities  for  socio-economic  business  concerns  with 
shorter,  simpler  solicitations  that  are  more  easily  understood  and  by 
allowing  any  interested  companies  to  respond  versus  IDVs,  which  are 
more  exclusive. 

3.  The  promotion  of  efficiency  and  economy  in  contracting  through  a 
streamlined  evaluation  process  and  the  ability  to  use  innovative 
approaches,  such  as  reverse  auctions. 

4.  Avoiding  unnecessary  burdens  for  agencies  and  contractors  by  allowing 
the  use  of  RFQs,  which  are  generally  less  burdensome  while  still 
permitting  the  use  of  RFPs  and  IFBs  if  the  situations  warrants. 

Generally,  if  a  process  is  simpler  and  less  time  consuming,  individuals  should 
naturally  be  expected  to  embrace  and  utilize  such  a  practice;  however,  despite  being 
established  to  extend  the  benefits  of  SAP  to  a  larger  number  of  commercial  item 
acquisitions,  that  does  not  appear  to  have  occurred  in  regard  to  the  Test  Program  for 
Certain  Commercial  Items.  The  results  of  this  research  culminated  in  the  following 
answers  to  the  research  questions: 
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1.  What  was  the  Air  Force’s  usage  rate  of  FAR  13,5  Test  Program  for  Certain 

Commercial  Items  procedures  from  February  1,  2013,  through  May  31, 

2014? 

After  factoring  out  alternate  procurement  methods  (such  as  ID  Vs)  that  may  be 
either  required  or  viewed  by  a  contracting  official  as  equally  or  more  efficient,  the  usage 
rate  of  the  test  program  in  the  Air  Force  is  still  lagging  somewhere  between  27.79%  and 
51.49%  of  all  eligible  contract  actions.  In  other  words,  approximately  half  to  three- 
quarters  of  Air  Force  commercial  acquisitions  between  $150,000  and  $6,500,000 
needlessly  subject  themselves  to  more  complex,  exhaustive  procedures  in  order  to  acquire 
goods  and  services  that  are  predominantly  available  in  the  commercial  market. 

2.  What  factors  may  he  limiting  the  use  of  FAR  13.5  Commercial-Item  Test 

Program  procedures? 

Some  likely  contributors  to  the  low  usage  rate  of  the  test  program  in  the  Air  Force 
were  the  following: 

•  The  temporary  nature  of  the  Test  Program  for  Certain  Commercial  Items. 

•  The  absence  of  guidance  within  the  DOD  and  AF  to  encourage  or  require 
the  test  program’s  use. 

•  The  lack  of  training  and  educational  materials  on  test  program  procedures. 

Another  factor  impacting  the  ability  to  definitively  assess  the  usage  rate  of  the  test 
program  within  the  Air  Force  was  the  inconsistency  in  coding  contract  actions  in  FPDS- 
NG.  In  order  to  alleviate  or  eliminate  these  issues,  this  paper  makes  the  following 
rec  ommendations . 

C.  RECOMMENDATIONS 

1.  Recommendation  1 — Permanently  Authorize  the  Test  Program  for 
Certain  Commercial  Items 

The  Test  Program  for  Certain  Commercial  Items  has  been  in  existence  for 
approximately  18  years  from  the  time  it  was  originally  enacted  under  FARA;  however, 
during  that  time,  the  program  has  remained  temporary,  requiring  reauthorization  by 
Congress  each  time  it  expired.  During  the  GAO’s  research  into  the  usage  of  the  test 
program,  the  “DHS  and  DOI  suggested  that  the  temporary  nature  of  the  test  program 
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hindered  its  use  and  reeommended  that  the  authority  be  made  permanent”  (GAO,  2014, 
p.  23). 

When  this  researeh  began,  the  initial  version  of  the  NDAA  for  FY  2015,  H.R. 
4435,  dated  May  13,  2014,  did  not  eontain  a  provision  to  extend  the  authority  of  the  test 
program;  however.  Congressman  Gerry  Connolly,  representing  the  11th  distriet  of 
Virginia,  proposed  an  amendment  “to  make  the  authority  to  use  simplified  aequisition 
proeedures  for  eertain  eommereial  items  permanent”  (House  of  Representatives, 
Committee  on  Rules,  2014,  Arndt.  204.  As  of  September  29,  2014,  the  eurrent  version  of 
the  NDAA  for  FY  2015,  dated  June  5,  2014,  whieh  was  on  the  Senate’s  ealendar  for 
eonsideration,  would,  if  passed  by  the  Senate  and  signed  by  the  President,  make  the  Test 
Program  for  Certain  Commereial  Items  permanent  by  amending  “Seetion  4202  of  the 
Clinger-Cohen  Aet  of  1996  (division  D  of  Publie  Law  104-106;  10  U.S.C.  2304  note)”  to 
strike  subseetion  (e)  (NDAA  for  FY  2015,  2014,  p.  277).  The  results  of  this  researeh 
strongly  support  this  ehange  to  the  test  program  in  order  to  eliminate  the  previous 
tenuous,  unpredietable  status  of  its  authority. 

An  additional  potential  benefit  of  making  the  test  program  permanent  was 
identified  when 

the  Offiee  of  Management  and  Budget  noted  that  in  the  event  that  the 
authority  was  made  permanent,  it  would  expeet  ageneies  to  periodieally 
review  their  use  of  the  test  program  as  part  of  their  management  reviews. 
Consequently,  we  believe  that  eolleeting  and  assessing  data  on  the 
program’s  use  would  be  benefieial.  (GAO,  2014,  p.  23) 

2,  Recommendation  2 — Incorporate  FAR  13,5  Usage  Provisions  Into  the 

AFICA  MP 

The  Air  Foree  Installation  Contraeting  Ageney  (AFICA)  should  ineorporate  FAR 
13.5  usage  provisions  into  their  mandatory  proeedures  to  eneourage  or  require  the  use  of 
the  test  program  for  eligible  aequisitions.  This  requirement  would  not  supersede  other 
mandatory  souree  or  disallow  the  use  of  an  available  IDV  if,  in  the  judgment  of  the 
eontraeting  professional,  an  alternate  method  was  the  better  proeurement  option; 
however,  those  48.51%  to  72.21%  of  eontraet  notions  that  use  more  oomplex,  exhaustive 
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procedures  with  no  justifiable  reason  should  be  reviewed  and  required  to  use  the  more 
effieient  and  eeonomieal  approaeh  provided  by  the  test  program. 

AFICA’s  MP  states  that  their  organization  is  responsible  for  reviewing  all 
eontraet  aetions  in  exeess  of  $2,000,000  (AFICA,  2013).  During  the  period  analyzed  by 
this  researeh,  189  eontraet  aetions  valued  at  $2,000,000  or  higher  did  not  use  test 
program  proeedures.  If  AFICA  had  required  those  organizations  to  use  the  test  program 
on  those  requirements  as  part  of  their  review  that  would  have  single-handedly  inereased 
the  usage  rate  by  over  10%.  Moreover,  it  is  plausible  that  these  proeedures,  if  enforeed  by 
AFICA  on  aequisitions  $2,000,000  and  higher,  would  triekle  down  within  ageneies  to 
eligible  aetions  of  lower  dollar  amounts,  further  inereasing  the  Air  Foree’s  usage  rate. 

3.  Recommendation  3 — Incorporate  FAR  13.5  Into  DAU  SAP  Training 

Even  if  Congress  makes  the  Test  Program  for  Certain  Commereial  Items 
permanently  available  and  AFICA  requires  the  program’s  usage,  eontraeting 
professionals  must  understand  the  proeedures  in  order  to  eorreetly  and  effeetively  employ 
them.  Therefore,  the  DAU  should  ineorporate  the  test  program  methodology  and 
proeedures  into  its  CON  237  “Simplified  Aequisition  Proeedures”  training  module  and 
ereate  an  additional  Continuous  Learning  Module  speeifieally  for  the  use  of  SAP  in 
exeess  of  the  SAT  as  authorized  by  FAR  13.5. 

4.  Recommendation  4 — Update  FPDS-NG  to  Prevent  Coding 
Inconsistencies 

In  order  to  improve  the  aeeuraey  of  test  program  data,  FPDS-NG  should  be 
updated  by  the  General  Serviees  Administration  to  eorreet  the  eurrent  eoding 
ineonsisteney  identified  by  this  researeh  between  Data  Element  lOJ  “Commereial  Item 
Test  Program”  and  Data  Element  lOA  “Extent  Competed.”  Currently,  the  system 
prevents  aetions  eoded  under  Data  Element  lOJ  as  having  utilized  the  test  program  from 
ehoosing  any  eompetitive  proeess  besides  “eompeted  under  SAP”  and  “not  eompeted 
under  SAP.”  Conversely,  the  update  should  prevent  eontraet  aetion  reports  (CARs)  from 
finalizing  if  a  non-eontingeney  aetion  in  exeess  of  the  SAT  is  eoded  under  Data  Element 
lOA  as  either  “eompeted  under  SAP”  or  “not  eompeted  under  SAP,”  but  not  eoded  as 
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using  the  test  program  under  Data  Element  lOJ  (sinee  the  authority  to  use  SAP  in  exeess 
of  $150,000  eomes  from  the  test  program). 


D.  AREAS  FOR  FURTHER  INVESTIGATION  AND  RESEARCH 

This  research  can  be  expanded  and  improved  upon  in  several  ways.  The  following 
list  includes  areas  for  additional  research  opportunities: 

1 .  Researchers  could  examine  random  samples  of  eligible  contract  files  to 
identify  any  potential  issues  with  under  or  over  reporting.  Physically 
examining  contract  files  would  also  be  useful  to  identify  any  best  practices 
used  by  contracting  professionals  in  employing  the  authority  of  the  test 
program. 

2.  Researchers  could  survey  and  interview  operational  contracting  officials  to 
obtain  qualitative  data  on  the  test  program.  The  survey  should  focus  on 
randomly  sampling  acquisition  professionals  in  operational  contracting 
squadrons  since  those  acquisitions  are  most  likely  to  be  eligible  for  the  test 
program  (i.e.,  commercial  items  between  $150,000  and  $6,500,000).  The 
survey  should  seek  to  obtain  information  that  assesses  contracting 
professionals’  awareness,  understanding,  and  application  of  test  program 
procedures. 

3.  Researchers  could  survey  and  interview  contractors  to  obtain  qualitative 
data  on  the  test  program  from  an  industry  perspective.  The  survey  should 
focus  on  assessing  their  preference  and  competence  when  it  comes  to 
solicitations  under  FAR  13.5  versus  more  lengthy  and  complex 
solicitations  under  FAR  Part  14  or  15. 

E.  CHAPTER  SUMMARY 

This  chapter  concluded  the  research  into  the  Air  Force’s  usage  rate  of  the  FAR 
Subpart  13.5  Test  Program  for  Certain  Commercial  Items.  Multiple  recommendations 
were  made  to  improve  the  usage  of  the  test  program  or  the  ability  to  analyze  that  usage. 
Finally,  future  researchers  curious  about  the  test  program’s  effectiveness  and  benefits 
were  provided  with  a  few  areas  worthy  of  additional  investigation. 
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